Reinventing TOTP: A Journey from First Principles

Table of Contents

1. The Problem We're Trying to Solve
2. The One-Time Password (OTP)
3. Exploring Solutions: Generating Random Passwords Independently
4. Time as the Variable (TOTP)
5. Handling Clock Skew and Time Steps
6. The Complete Algorithm
7. Why This Design Works
8. Downsides and Implementation Considerations
9. Security Considerations
10. The Evolution
11. Conclusion

The Problem We're Trying to Solve

Imagine you're building a system where users need to authenticate themselves. The traditional approach is simple: username and password. But passwords have several fundamental problems:

• Weak passwords can be guessed: Simple passwords like "123456" or "password" are easily guessed by attackers.
• Can be stolen: Passwords can be intercepted through phishing attacks, keyloggers, or social engineering.
• Server-side breaches: Databases get hacked, and password hashes (or worse, plaintext passwords) get leaked.
• Brute force attacks: Weak passwords can be cracked through automated brute force attempts.
• Password reuse: Users often reuse the same password across multiple sites, so one breach compromises many accounts.
• No expiration: Once stolen, a password remains valid indefinitely until manually changed.
• Human factor: Users tend to choose weak, memorable passwords that are easy to crack.

Once an attacker has your password, they can impersonate you indefinitely—until you discover the breach and change it (if you even know it was compromised).

The core problem: How do we create a credential that's valid only once, or only for a short period of time?

This is the foundation of multi-factor authentication (MFA). Even if someone steals your password, they can't use it without also having access to your second factor—something that changes frequently and can't be easily reused.

So you start to think about coming up with a solution. You soon realize the system must have the following properties at the minimum:

1. Should be random: Each password must appear random and unpredictable to an observer. Even if someone sees one password, they shouldn't be able to predict the next one.

1. Cannot be guessed or brute forced: The password space must be large enough that an attacker can't feasibly try all possibilities within the valid time window. Additionally, the generation algorithm must be cryptographically secure.

1. Should be one-time use only: Once a password is used, it must become invalid. This prevents replay attacks where an attacker intercepts and reuses a password.

1. Inexpensive and ideally must work offline: The solution shouldn't require expensive infrastructure (like SMS gateways) or constant network connectivity. Users should be able to authenticate even when they're offline or in areas with poor connectivity.

The One-Time Password (OTP)

Let's start with the simplest solution: one-time passwords.

The idea is straightforward:

1. Generate a random password
2. Send it to the user (via SMS, email, etc.)
3. The user enters it to authenticate
4. Once used, it's invalid forever

While this works, you soon realize this has several problems:

• Problems:
1. Requires communication: This requires communication with the user every time they want to log in. What if they're offline? What if SMS is delayed? What if they're in an area with no cell service?
2. Delivery cost via platform: Sending OTPs via SMS or email incurs costs. SMS gateways charge per message, and email services have costs at scale. These costs add up quickly, especially for high-traffic applications with frequent logins.
3. Server-side storage cost: The server must store each generated OTP until it's used or expires. This adds storage overhead and cost, especially at scale.
4. Invalidation complexity: If a new OTP is generated before the old one is used, the server must invalidate the previous one. This requires state management and can lead to race conditions or confusion if multiple OTPs are in flight.
5. Interception risk: Someone can eavesdrop on the network or could get access to your device.

We need something that works offline and doesn't require a round-trip to a server, and doesn't require the server to store any OTP state.

Exploring Solutions: Generating Random Passwords Independently

You think of this hard and soon enough you land on the insight: What if both the server and the client could independently generate the same password?

The first thought would be to generate one-time use values that are synced between client and server independently, without requiring network communication. If both sides can arrive at the same value at the same time, we've solved the synchronization problem.

Attempt 1: Pre-generated OTP Lists

The most straightforward approach might be: store a list of OTPs on both the server and client.

Server stores: [123456, 789012, 345678, ...]
Client stores: [123456, 789012, 345678, ...]

User logs in → Client sends first OTP → Server checks if it matches first in list

• Pitfalls:
• Storage overhead: Both sides need to store potentially thousands of OTPs
• Synchronization nightmare: What if the client uses OTP #5 but the server thinks we're on OTP #3? They're now permanently out of sync
• Limited lifespan: Once you run out of pre-generated OTPs, you need to regenerate and sync the entire list again
• Security risk: If either side is compromised, the attacker has access to all future OTPs
• Multi-device problem: If a user has multiple devices, each needs its own list, or they need to stay perfectly synchronized

This approach doesn't scale and introduces more problems than it solves.

Attempt 2: Random Numbers

First thought: What if we use random numbers?

Client generates random: 42
Server generates random: 42  ← Wait, how does this work?

• Problems:
• Synchronization impossible: How does the server know which random number the client generated? They're random!
• No shared state: Random number generators on different devices produce different sequences
• Requires communication: To sync random numbers, you'd need to communicate them, which defeats the purpose
• Predictability concerns: If the random number generator is predictable, security is compromised

Random numbers don't work because there's no way for both sides to independently arrive at the same random value.

• What about Pseudo-Random Number Generation (PRNG)?

You might think: "What if we use a pseudo-random number generator with a seed? Both sides could use the same seed and get the same sequence!"

Both sides use seed = 12345
PRNG(seed) → 42
PRNG(seed) → 17
PRNG(seed) → 99

• Problems with PRNG:
1. Seed synchronization: How do both sides know to use the same seed? You'd need to communicate the seed, which brings back the communication problem.
1. State synchronization: PRNGs maintain internal state. If the client generates 5 numbers but only uses the 3rd one, the PRNG state on both sides is now different. They're out of sync.
1. Predictability: PRNGs are deterministic algorithms. If an attacker can observe a few outputs, they might be able to predict future values or reverse-engineer the seed. This is a serious security vulnerability.
1. Cryptographic weakness: Most PRNGs (like those used in programming languages) are not cryptographically secure. They're designed for statistical randomness, not security. An attacker could potentially predict the sequence.
1. Seed management: You'd need to securely share and manage seeds, which is essentially the same problem as sharing a secret key—but PRNGs are less secure than cryptographic hash functions.
1. No forward secrecy: Once the seed is compromised, all past and future values can be calculated. Unlike time-based systems where each time step is independent, PRNG sequences are linked.

PRNGs don't solve the fundamental problem: you still need synchronized state, and they introduce additional security risks compared to cryptographic hash functions.

Attempt 3: Sequential Counter

Okay, random doesn't work. What if we use a sequential counter?

First, let's try using just the counter value itself as the password:

Both sides start at counter = 0
Client increments: counter = 1, password = "1"
Server increments: counter = 1, password = "1" ✓

• Problems with just counters:
• Predictable and sequential: The password is just the counter value. If someone sees password "5", they know the next one is "6". This is completely insecure.
• No randomness: The passwords are sequential numbers, making them trivial to guess.
• No cryptographic security: Anyone can predict future passwords just by knowing the pattern.
• Synchronization drift: If the client generates 5 OTPs (counter goes 0→5) but only uses the 3rd one, the server's counter is now at 1, but the client's is at 5. They're out of sync forever.
• Look-ahead complexity: The server needs to check multiple counter values (look-ahead window), but how many? What if the user generates 100 OTPs without using any?
• Multi-device chaos: If a user has the app on their phone and tablet, each device has its own counter. Using an OTP from one device desynchronizes the other.
• Replay window: Once you accept a counter value, how do you prevent it from being reused? You need to track which counters have been used, which brings back state management.

Counters alone don't work because they're predictable and lack security. But wait—what if we combine counters with hashing?

• The Hashing Solution:

So you think: How can one be generated at random, yet deterministically?

Being a brilliant engineer, you think: "If nothing works, throw hashing at the problem!"

You know that for a given input, a hash function always produces the same value. This is deterministic. But if you just hash the same key over and over, you'd get the same password every time—essentially the same as a static password. That's not secure.

Then the insight hits: What if we generate a sequence of values, and both the client and server hash the same value in the sequence? They would both arrive at the same hash result!

But what should this sequence be? What values should we use?

You realize: What if we hash the counter value with a secret key? This would make the output unpredictable while still being deterministic!

Both sides start at counter = 0
Both sides know: secret_key = "my-secret-key"
Client increments: counter = 1, hash(secret_key, 1) → "123456"
Server increments: counter = 1, hash(secret_key, 1) → "123456" ✓

This seems promising! Both sides can maintain a counter and increment it, and the hash function ensures the output is random and unpredictable.

Exploring Counter-Based Approach (HOTP)

Before we move to time, let's explore the counter-based approach more deeply. This is the basis of HOTP (HMAC-based One-Time Password).

The idea is simple: each time we generate a password, we increment a counter. The password is generated from both the secret and the counter.

password = generate(secret_key, counter)

• How it works:
1. Server and client share a secret key
2. Both maintain a counter (starting at 0)
3. To authenticate, client generates: HMAC-SHA1(secret, counter)
4. Converts the HMAC result to a 6-digit number
5. Server does the same calculation and compares
• The HMAC function:
• HMAC (Hash-based Message Authentication Code) ensures that even small changes to the input produce completely different outputs
• SHA-1 provides cryptographic security
• The result is deterministic: same inputs always produce same output
• Counter-based Flow Diagram:

┌────────────────────────────────────────────────────────────────┐
│              HOTP Authentication Flow                          │
├────────────────────────────────────────────────────────────────┤
│                                                                │
│  ┌──────────┐                     ┌──────────┐                 │
│  │  Client  │                     │  Server  │                 │
│  └────┬─────┘                     └────┬─────┘                 │
│       │                                │                       │
│       │ Both sides maintain counter    │                       │
│       │ (Initially synchronized)       │                       │
│       │ Client counter: 3              │                       │
│       │ Server counter: 3              │                       │
│       │                                │                       │
│       │ (Client generates 2 OTPs but   │                       │
│       │  doesn't use them - drift!)    │                       │
│       │ Client counter: 5              │                       │
│       │ Server counter: 3              │                       │
│       │                                │                       │
│       │ 1. Client increments counter   │                       │
│       │    counter = 5 + 1 = 6         │                       │
│       │                                │                       │
│       │ 2. Hash and generate 6 digit   │                       │
│       │    number → "456789"           │                       │
│       │                                │                       │
│       │ 3. Display to user             │                       │
│       │    "456789"                    │                       │
│       │                                │                       │
│       │ 4. User enters "456789" ──────>│                       │
│       │                                │                       │
│       │                                │ 5. Server checks      │
│       │                                │    counter 3, 4, 5,   │
│       │                                │    6, 7 (look-ahead)  │
│       │                                │                       │
│       │                                │ 6. Finds match at     │
│       │                                │    counter = 6        │
│       │                                │    → "456789" ✓       │
│       │                                │                       │
│       │                                │ 7. Update server      │
│       │                                │    counter to 6       │
│       │                                │                       │
│       │ 8. Authentication success <────│                       │
│       │                                │                       │
│  ┌─────────────────────────────────────┴───────────────────────┤
│  │                                                             │
│  │ Problem: If client generates OTPs but doesn't use them,     │
│  │          counters drift apart and sync is lost              │
│  └─────────────────────────────────────────────────────────────┘

• Remaining problems (even with hashing):
• Synchronization drift: If the client generates 5 OTPs (counter goes 0→5) but only uses the 3rd one, the server's counter is now at 1, but the client's is at 5. They're out of sync forever.
• Look-ahead complexity: The server needs to check multiple counter values (look-ahead window), but how many? What if the user generates 100 OTPs without using any?
• Multi-device chaos: If a user has the app on their phone and tablet, each device has its own counter. Using an OTP from one device desynchronizes the other.
• Replay window: Once you accept a counter value, how do you prevent it from being reused? You need to track which counters have been used, which brings back state management.

Counters have the same fundamental problem: they require state synchronization.

This is why we need a better solution—one that doesn't require maintaining synchronized counters.

The Light Bulb Moment: Time

Then you think: What is something that is constantly changing, but both the client and server can independently arrive at the correct value?

• Time! Like Bilbo Baggins figured out: Time is the answer.

Time is constantly changing, but both the client and server can independently ask "What time is it right now?" and get (approximately) the same answer. No synchronization needed. No state to maintain. No communication required.

Current time: 10:30:45
Both sides hash: hash(secret, current_time) → "456789"

Time naturally provides the sequence we need, and it's automatically synchronized. However, we'll need to handle some practical issues with clock differences, which we'll explore later.

Time as the Variable (TOTP)

Here's the elegant solution: Use time instead of a counter!

Time is naturally synchronized (well, mostly—we'll handle clock skew). Both the server and client can independently calculate: "What time is it right now?" and use that as the variable.

This is TOTP (Time-based One-Time Password).

• How it works:
1. Server and client share a secret key
2. Both get the current time (Unix timestamp)
3. Both calculate: HMAC-SHA1(secret, current_time), then truncate to 6 digits
4. Both arrive at the same password
• Example:

Current time: 2024-01-15 10:30:45 UTC
Unix timestamp: 1705315845

Both sides calculate:
HMAC-SHA1(secret, 1705315845) = [0x3f, 0x8a, 0x2c, ...]
(apply truncation algorithm)
Password: 456789

Both the client and server independently calculate the same password from the same secret and current time. No communication needed!

But wait—if we use the exact current time, the password changes every second! That's too frequent. Users need time to read and type the password. Also, we need passwords to expire after a reasonable period for security.

And there's another issue: when the client generates an OTP and sends it to the server, some time would have elapsed. The generated value at the server might be slightly off due to clock differences or network delay. How do you handle this?

Designing the Algorithm

Now you tinker with how the algorithm should work. You need three core components:

1. Shared Secret Key Or Initial Handshake: A secret that only the server and client know. This is established once during setup (via QR code, manual entry, etc.) and stored securely on both sides.
1. Hash Function: A cryptographic hash function that takes the secret and a time-based value, producing a deterministic but unpredictable output. HMAC-SHA1 is perfect for this—it's designed exactly for this use case (Hash-based Message Authentication Code).
1. Verification Process:
• Client calculates: HMAC(secret, time_value) → truncate to 6 digits
• Server calculates: HMAC(secret, time_value) → truncate to 6 digits
• Compare: if they match, authentication succeeds
• Handle clock differences: we'll need a mechanism to account for small time differences (we'll explore this in the next section)

Example Flow Diagram

Here's the complete flow from initial handshake to OTP use:

┌─────────────────────────────────────────────────────────────────┐
│              Phase 1: Initial Handshake (One Time)              │
├─────────────────────────────────────────────────────────────────┤
│                                                                 │
│  ┌──────────┐                     ┌──────────┐                  │
│  │  Client  │                     │  Server  │                  │
│  └────┬─────┘                     └────┬─────┘                  │
│       │                                │                        │
│       │                                │ 1. Generate secret     │
│       │                                │    secret_key =        │
│       │                                │    "JBSWY3DPEHPK3PXP"  │
│       │                                │                        │
│       │                                │ 2. Store secret_key    │
│       │                                │    securely in DB      │
│       │                                │                        │
│       │                                │ 3. Encode as QR code   │
│       │                                │    or display for      │
│       │                                │    manual entry        │
│       │                                │                        │
│       │ 4. User scans QR code or       │                        │
│       │    manually enters secret      │                        │
│       │                                │                        │
│       │ 5. Store secret_key securely   │                        │
│       │    on client device            │                        │
│       │                                │                        │
│       │ 6. Handshake complete ✓        │                        │
│       │    Both sides now share        │                        │
│       │    the same secret_key         │                        │
│       │                                │                        │
│       └────────────────────────────────┴─────────────────────┘  |
└─────────────────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────────────┐
│        Phase 2: OTP Generation (Client Side - Offline)      │
├─────────────────────────────────────────────────────────────┤
│                                                             │
│  ┌──────────┐                                               │
│  │  Client  │                                               │
│  └────┬─────┘                                               │
│       │                                                     │
│       │ 1. User opens authenticator app                     │
│       │                                                     │
│       │ 2. App retrieves stored secret_key                  │
│       │                                                     │
│       │ 3. Get current Unix timestamp                       │
│       │    current_time = 1705315845                        │
│       │                                                     │
│       │ 4. Calculate time step                              │
│       │    T = current_time // 30                           │
│       │    T = 56843861                                     │
│       │                                                     │
│       │ 5. Generate 6-digit OTP                             │
│       │    OTP = "456789"                                   │
│       │                                                     │
│       │ 6. Display OTP to user                              │
│       │    "456789" (valid for ~30 seconds)                 │
│       │                                                     │
│  └─────────────────────────────────────────────────────────┘|
└─────────────────────────────────────────────────────────────┘

┌───────────────────────────────────────────────────────────────┐
│     Phase 3: Authentication (User Login with OTP)             │
├───────────────────────────────────────────────────────────────┤
│                                                               │
│  ┌──────────┐                     ┌──────────┐                │
│  │  Client  │                     │  Server  │                │
│  └────┬─────┘                     └────┬─────┘                │
│       │                                │                      │
│       │ 1. User enters username        │                      │ 
│       │    and password                │                      │
│       │                                │                      │
│       │ 2. User enters OTP from        │                      │
│       │    authenticator app           │                      │
│       │    OTP = "456789"              │                      │
│       │                                │                      │
│       │ 3. Send login request ────────>│                      │
│       │    (username, password, OTP)   │                      │
│       │                                │                      │
│       │                                │ 4. Verify username   │
│       │                                │    and password      │
│       │                                │                      │
│       │                                │ 5. Retrieve user's   │
│       │                                │    secret_key from   │
│       │                                │    database          │
│       │                                │                      │
│       │                                │ 6. Get current time  │
│       │                                │    current_time =    │
│       │                                │    1705315845        │
│       │                                │                      │
│       │                                │ 7. Calculate time    │
│       │                                │    step T            │
│       │                                │    T = 56843861      │
│       │                                │                      │
│       │                                │ 8. Check time window │
│       │                                │    (T-1, T, T+1)     │
│       │                                │    to handle clock   │
│       │                                │    skew              │
│       │                                │                      │
│       │                                │ 9. For each T in     │
│       │                                │    window:           │
│       │                                │    - Calculate HMAC  │
│       │                                │    - Convert to 6    │
│       │                                │      digits          │
│       │                                │    - Compare with    │
│       │                                │      user's OTP      │
│       │                                │                      │
│       │                                │ 10. Match found! ✓   │
│       │                                │     OTP is valid     │
│       │                                │                      │
│       │ 11. Authentication success <───│                      │
│       │     User logged in             │                      │
│       │                                │                      │
│       └────────────────────────────────┴──────────────────────┤
│                                                               │
│  Note: OTP expires after time step (30 seconds). Server       │
│        accepts OTPs from T-1, T, and T+1 to handle clock      │
│        skew and network delays.                               │
└───────────────────────────────────────────────────────────────┘

Handling Clock Skew and Time Steps

You realize there are two problems to solve:

1. The password changes too frequently (every second)
2. Time differences between client and server, plus network delays
• The Time Step Solution:

You realize: What if we divide time into steps? Instead of using the exact current time, we divide time into intervals (time steps). The password stays the same for the duration of each step, then changes to a new value.

For example, with a 30-second time step:

• From 10:30:00 to 10:30:29 → same password
• From 10:30:30 to 10:30:59 → new password
• From 10:31:00 to 10:31:29 → another new password

This gives users enough time to read and enter the password, while still keeping it secure with regular expiration.

• Why 30 seconds?
• Short enough to be secure (password expires quickly)
• Long enough for users to type it in comfortably
• Balances security and usability
• The Algorithm with Time Steps:

Now the algorithm becomes:

1. Server and client share a secret key
2. Both calculate the current time step: T = floor((current_time - T0) / X)
• T0 is the Unix epoch (January 1, 1970, 00:00:00 UTC)
• X is the time step (typically 30 seconds)
1. Generate password: HMAC-SHA1(secret, T), then truncate to 6 digits
2. The password is valid for the duration of the time step
• Example:

Current time: 2024-01-15 10:30:45 UTC
Unix timestamp: 1705315845
Time step (X): 30 seconds
T0: 0 (Unix epoch)

T = floor((1705315845 - 0) / 30) = 56843861

HMAC-SHA1(secret, 56843861) = [0x3f, 0x8a, 0x2c, ...]
(apply truncation algorithm)
Password: 456789

This password is valid from 10:30:30 to 10:31:00.

• But there's still the second problem: time differences and network delays
• Clock skew: The client's clock and server's clock might not be perfectly synchronized. One might be a few seconds ahead or behind.
• Network delay: When the client generates the OTP, it's based on the client's current time. But by the time the server receives and verifies it, some time has passed. If we're at the boundary of a time step, this could cause a mismatch.

So you think: how would you handle this? Clearly there would be some sync delay. The time step helps, but we need an additional mechanism.

• Solution: Add a buffer window option.

Instead of only accepting the exact current time step, accept a small window around it. This accounts for:

• Clock skew between client and server
• Network transmission delay
• Processing time

Instead of only accepting the current time step (T), accept:

• Previous time step (T-1) - in case client clock is slightly behind or there's network delay
• Current time step (T) - the ideal case
• Next time step (T+1) - in case client clock is slightly ahead

This gives us a buffer window that handles typical clock skew and network delays while maintaining security. The window size is configurable—typically ±1 time step, which with a 30-second time step gives us a 90-second acceptance window (3 × 30 seconds).

Now the verification process becomes:

• Client calculates: HMAC(secret, current_time_step) → truncate to 6 digits
• Server calculates: HMAC(secret, current_time_step) → truncate to 6 digits
• Server also checks adjacent time steps (T-1, T, T+1) to handle clock skew
• Compare: if any match, authentication succeeds

Making It Flexible

Being a great engineer, you decide to make things flexible. What if different people need different algorithms or different digit lengths?

• Configurable parameters:
• Hash algorithm: HMAC-SHA1 (default), but could support HMAC-SHA256, HMAC-SHA512 for stronger security
• Digit length: 6 digits (default), but could be 4, 6, 7, or 8 digits depending on security vs. usability trade-offs
• Time step: 30 seconds (default), but could be 15, 30, or 60 seconds
• Clock skew window: ±1 time step (default), but adjustable based on expected clock accuracy

This flexibility allows the system to adapt to different security requirements and use cases.

The Complete Algorithm

Here's the complete TOTP algorithm:

import hmac
import hashlib
import time
import struct

def convert_to_digits(hmac_result, digits=6):
    """
    Convert HMAC result to a specified number of digits.
    
    Args:
        hmac_result: HMAC digest (bytes)
        digits: Number of digits desired (default 6)
    
    Returns:
        OTP as string with specified number of digits
    """
    # Dynamic truncation algorithm
    offset = hmac_result[-1] & 0x0F  # Last 4 bits
    binary = struct.unpack('>I', hmac_result[offset:offset+4])[0]
    binary = binary & 0x7FFFFFFF  # Mask MSB
    
    # Convert to digits
    otp = binary % (10 ** digits)
    
    return str(otp).zfill(digits)

def generate_totp(secret_key, time_step=30, digits=6):
    """
    Generate a TOTP password.
    
    Args:
        secret_key: Shared secret (bytes)
        time_step: Time step in seconds (default 30)
        digits: Number of digits in password (default 6)
    
    Returns:
        TOTP password as string
    """
    # Calculate current time step
    current_time = int(time.time())
    T = current_time // time_step
    
    # Convert T to bytes (8 bytes, big-endian)
    T_bytes = struct.pack('>Q', T)
    
    # Calculate HMAC-SHA1
    hmac_result = hmac.new(secret_key, T_bytes, hashlib.sha1).digest()
    
    # Convert to digits
    return convert_to_digits(hmac_result, digits)

def verify_totp(secret_key, user_input, time_step=30, digits=6, window=1):
    """
    Verify a TOTP password with clock skew tolerance.
    
    Args:
        secret_key: Shared secret (bytes)
        user_input: Password entered by user
        time_step: Time step in seconds (default 30)
        digits: Number of digits in password (default 6)
        window: Number of time steps to check on each side (default 1)
    
    Returns:
        True if valid, False otherwise
    """
    current_time = int(time.time())
    current_T = current_time // time_step
    
    # Check current time step and window around it
    for i in range(-window, window + 1):
        T = current_T + i
        T_bytes = struct.pack('>Q', T)
        hmac_result = hmac.new(secret_key, T_bytes, hashlib.sha1).digest()
        
        otp = convert_to_digits(hmac_result, digits)
        
        if otp == user_input:
            return True
    
    return False

Why This Design Works

1. Offline capability: No network communication needed after initial secret sharing
2. Time synchronization: Uses time, which is naturally synchronized (with skew tolerance)
3. Cryptographic security: HMAC-SHA1 ensures passwords can't be predicted or reverse-engineered
4. Short validity window: 30-second expiration limits attack window
5. Simple for users: Just 6 digits to type
6. Deterministic: Same inputs always produce same output (critical for verification)

Downsides and Implementation Considerations

While TOTP is elegant, it's not without its challenges. Let's explore the downsides and important implementation details.

Downsides of TOTP

1. Clock synchronization dependency: TOTP relies on both client and server having reasonably synchronized clocks. If a device's clock is significantly off (hours or days), TOTP won't work. Users must ensure their device time is set correctly.
1. Single point of failure: If the secret key is compromised on either the client or server side, the entire security mechanism is broken. An attacker with the secret can generate valid OTPs indefinitely.
1. No revocation mechanism: Once a secret is shared, there's no built-in way to revoke it without user intervention. If a device is lost or compromised, the user must manually disable TOTP and re-enroll.
1. Limited to 6-8 digits: The typical 6-digit OTP provides 1 million possible values. While the time window limits brute force attacks, it's still a relatively small search space compared to longer passwords.
1. User experience friction: Users must have their authenticator device available and working. If the device is lost, broken, or the battery is dead, they can't authenticate.
1. Backup and recovery complexity: If users lose access to their authenticator app, recovery can be difficult. Services typically provide backup codes, but these must be stored securely by users.

How the Server Should Store Secrets

The security of TOTP depends critically on how the server stores the secret keys. Here are the best practices:

• 1. Encryption at Rest:
• Secrets should never be stored in plaintext in the database
• Use strong encryption (AES-256) to encrypt secrets before storing them
• Store encryption keys separately from the database (use a key management service or hardware security module)
• 2. Secure Storage:

❌ Bad: Storing plaintext
Database: user_id=123, totp_secret="JBSWY3DPEHPK3PXP"

✅ Good: Encrypted storage
Database: user_id=123, totp_secret_encrypted="aGVsbG8gd29ybGQ="
Encryption key: Stored in secure key management service

• 3. Access Control:
• Limit database access to only necessary services
• Use principle of least privilege
• Log all access to secret data for audit purposes
• Implement database-level encryption and access controls
• 4. Key Management:
• Use a dedicated key management service (AWS KMS, Azure Key Vault, HashiCorp Vault)
• Rotate encryption keys periodically
• Never hardcode encryption keys in application code
• Use environment variables or secure configuration management
• 5. Secure Transmission:
• When sharing the secret initially (via QR code or manual entry), ensure the connection is encrypted (HTTPS)
• Never send secrets via email or SMS in plaintext
• Use secure channels for all secret transmission
• 6. Backup and Recovery:
• Encrypt backups of the database
• Store backup encryption keys separately
• Test recovery procedures regularly
• Consider using database replication with encryption
• Example Secure Storage Pattern:

# Server-side secret storage (pseudocode)
def store_totp_secret(user_id, secret):
    # Encrypt the secret using a key from key management service
    encryption_key = get_key_from_kms()
    encrypted_secret = encrypt(secret, encryption_key)
    
    # Store encrypted secret in database
    database.store(user_id, encrypted_secret)
    
    # Never log the plaintext secret
    log.info(f"Stored encrypted TOTP secret for user {user_id}")

def retrieve_totp_secret(user_id):
    # Retrieve encrypted secret from database
    encrypted_secret = database.retrieve(user_id)
    
    # Decrypt using key from key management service
    encryption_key = get_key_from_kms()
    secret = decrypt(encrypted_secret, encryption_key)
    
    return secret

• 7. Compliance Considerations:
• Follow industry standards (PCI DSS, GDPR, HIPAA) for sensitive data storage
• Implement proper data retention and deletion policies
• Ensure secrets are properly destroyed when no longer needed
• Maintain audit logs for compliance

Remember: The security of TOTP is only as strong as the security of the secret storage mechanism. A compromised database with plaintext secrets renders the entire TOTP system useless.

Security Considerations

1. Secret key protection: The secret must be stored securely on both ends
2. Replay attacks: Prevented by short time window
3. Brute force: 6 digits = 1 million possibilities, but 30-second window makes brute force impractical
4. Clock skew: Handled by acceptance window
5. Secret sharing: Initial secret must be shared securely (QR code, manual entry, etc.)

The Evolution

We've journeyed from:

• Static passwords → Reusable, vulnerable
• One-time passwords via communication → Secure but requires connectivity
• HOTP (counter-based) → Offline but synchronization issues
• TOTP (time-based) → Offline, naturally synchronized, elegant solution

Conclusion

TOTP is a beautiful example of solving a complex security problem with elegant simplicity. By using time as a naturally synchronized variable, we've created a system that:

• Works offline
• Is easy for users
• Provides strong security
• Requires minimal infrastructure

The next time you use Google Authenticator, Authy, or any TOTP app, you're witnessing this elegant solution in action. A shared secret, the current time, and a cryptographic hash function—that's all it takes to create a secure, one-time password system.

---

• This blog post walked through the thought process of reinventing TOTP from first principles. The actual TOTP standard (RFC 6238) builds on HOTP (RFC 4226) and includes additional considerations for implementation, but the core concepts remain the same.*
• Note: We've intentionally skipped some complexities and simplified certain aspects of the solution to make it easier to understand the core concepts. Real-world implementations include additional considerations such as proper secret encoding (Base32), more sophisticated truncation algorithms, support for different hash algorithms, and various edge cases in clock synchronization. However, the fundamental principles and thought process outlined here remain the foundation of how TOTP works.