Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions ps-cache-kotlin/Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
FROM maven:3.9-eclipse-temurin-21 AS builder
WORKDIR /app
COPY pom.xml .
RUN mvn dependency:go-offline -q
COPY src/ src/
RUN mvn package -DskipTests -q

FROM eclipse-temurin:21-jre-alpine
WORKDIR /app
COPY --from=builder /app/target/kotlin-app-1.0.0.jar app.jar
EXPOSE 8080
CMD ["java", "-jar", "app.jar"]
122 changes: 122 additions & 0 deletions ps-cache-kotlin/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,122 @@
# PS-Cache Kotlin — JDBC Prepared Statement Cache Mock Mismatch Reproduction

This sample demonstrates a bug in Keploy's Postgres mock matcher where **JDBC prepared statement caching combined with connection pool eviction causes the replay to return the wrong person's data**.

## The Bug

The JDBC driver (PostgreSQL JDBC + HikariCP) caches prepared statements per connection. When the connection pool evicts and creates a new connection, the PS cache is cold — but the recorded mocks from the evicted connection had warm-cache structure (Bind-only, no Parse). During replay, the matcher can't distinguish between mocks from different connection windows because:

1. All mocks have the same parameterized SQL: `SELECT ... WHERE member_id = ?`
2. `bindParamMatchLen` mode only checks parameter byte-length (all int4 are 4 bytes)
3. Sort-order prediction starts from 0 on a fresh connection, pointing to the wrong window's mocks

### Real-world impact
This was reported by a customer running a Kotlin/Spring Boot app with Agoda's travel account service. Test-6 (member_id=31) returned Alice's data (member_id=19) instead of Charlie's — **silently returning the wrong customer's financial data**.

## Architecture

```
┌──────────────────────┐
HTTP requests ──> │ Kotlin + Spring Boot │
│ HikariCP pool=1 │
│ prepareThreshold=1 │
└──────────┬───────────┘
┌──────────────────┼──────────────────┐
│ │ │
/account /evict /account
member=19 (pool evict) member=31
│ │ │
Connection A destroyed Connection B
PS cache: cold→warm PS cache: cold
│ │
1st: Parse+Bind+Desc+Exec Parse+Bind+Desc+Exec
2nd: Bind+Exec (cached PS)
│ │
mocks connID=0 mocks connID=2
(Alice, 1000) (Charlie, 500)
```

## How to Reproduce the Bug

### Prerequisites
```bash
docker run -d --name pg-demo -e POSTGRES_PASSWORD=testpass -e POSTGRES_DB=demodb -p 5433:5432 postgres:16
```

### Pre-create the schema
```bash
docker exec pg-demo psql -U postgres -d demodb -c "
CREATE SCHEMA IF NOT EXISTS travelcard;
CREATE TABLE IF NOT EXISTS travelcard.travel_account (
id SERIAL PRIMARY KEY, member_id INT NOT NULL UNIQUE,
name TEXT NOT NULL, balance INT NOT NULL DEFAULT 0);
INSERT INTO travelcard.travel_account (member_id, name, balance) VALUES
(19, 'Alice', 1000), (23, 'Bob', 2500),
(31, 'Charlie', 500), (42, 'Diana', 7500);"
```

### Build the app
```bash
mvn package -DskipTests -q
```

### With the OLD keploy binary (demonstrates failure)
```bash
# Record
sudo keploy record -c "java -jar target/kotlin-app-1.0.0.jar"
# Hit endpoints:
curl http://localhost:8090/account?member=19
curl http://localhost:8090/account?member=23
curl http://localhost:8090/evict
curl http://localhost:8090/account?member=31
curl http://localhost:8090/account?member=42
# Stop recording (Ctrl+C)

# Reset DB and replay
docker exec pg-demo psql -U postgres -d demodb -c "TRUNCATE travelcard.travel_account; INSERT INTO ..."
sudo keploy test -c "java -jar target/kotlin-app-1.0.0.jar" --skip-coverage
```

**Expected failure (without fix):**
```
test-5 (/account?member=31):
EXPECTED: {"memberId":31, "name":"Charlie", "balance":500}
ACTUAL: {"memberId":19, "name":"Alice", "balance":1000} ← WRONG PERSON
```

**With obfuscation enabled (worse):**
```
test-5: EXPECTED Charlie → ACTUAL Alice
test-6: EXPECTED Diana → ACTUAL Bob ← TWO wrong results
```

### With the FIXED keploy binary
```bash
# Same steps → all tests pass, correct data for each member
```

## What the Fix Does

The fix adds **recording-connection affinity** to the Postgres mock matcher (see [keploy/integrations#121](https://github.com/keploy/integrations/pull/121)):

1. When the first `Bind` mock is consumed on a replay connection, its recording `connID` is stored
2. Subsequent scoring applies +50/-50 bonus/penalty to prefer mocks from the same recording window
3. Only activates when 2+ distinct recording connections exist (zero impact on single-connection apps)

## Configuration

### application.properties
| Property | Value | Purpose |
|----------|-------|---------|
| `spring.datasource.hikari.maximum-pool-size` | `1` | Forces all requests through one connection |
| `prepareThreshold=1` | JDBC URL param | Caches PS after first use |
| `spring.sql.init.mode` | `never` | Schema created externally |

### Endpoints

| Endpoint | Description |
|----------|-------------|
| `GET /health` | Health check |
| `GET /account?member=N` | Query travel_account by member_id (BEGIN → SELECT → COMMIT) |
| `GET /evict` | Soft-evict HikariCP connections (forces new PG connection) |
34 changes: 34 additions & 0 deletions ps-cache-kotlin/docker-compose.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
services:
db:
image: postgres:16-alpine
environment:
POSTGRES_USER: postgres
POSTGRES_PASSWORD: postgres
POSTGRES_DB: testdb
ports:
- "5433:5432"
Copy link

Copilot AI Apr 16, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The DB is published on host port 5433, but the app defaults to DB_PORT=5432. If you expect running the app locally against the compose Postgres (without the api service), consider either mapping 5432:5432 or documenting that local runs need DB_PORT=5433 to avoid connection confusion.

Suggested change
- "5433:5432"
- "5432:5432"

Copilot uses AI. Check for mistakes.
Copy link

Copilot AI Apr 16, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

db is published on host port 5433, but the app’s default DB_PORT in application.properties is 5432. If someone runs Postgres via this compose file and runs the app locally (not in the api container), the default connection settings will point to the wrong port. Consider either mapping 5432:5432 here, or changing the default DB_PORT to 5433 (and documenting the intended run mode).

Suggested change
- "5433:5432"
- "5432:5432"

Copilot uses AI. Check for mistakes.
volumes:
- pgdata:/var/lib/postgresql/data
- ./init.sql:/docker-entrypoint-initdb.d/init.sql
healthcheck:
test: ["CMD-SHELL", "pg_isready -U postgres"]
interval: 2s
timeout: 5s
retries: 5

api:
build: .
ports:
- "8080:8080"
environment:
DB_HOST: db
DB_PORT: "5432"
DB_USER: postgres
DB_PASSWORD: postgres
DB_NAME: testdb
depends_on:
db:
condition: service_healthy

volumes:
pgdata:
15 changes: 15 additions & 0 deletions ps-cache-kotlin/init.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
CREATE SCHEMA IF NOT EXISTS travelcard;

CREATE TABLE IF NOT EXISTS travelcard.travel_account (
id SERIAL PRIMARY KEY,
member_id INT NOT NULL UNIQUE,
name TEXT NOT NULL,
balance INT NOT NULL DEFAULT 0
);

INSERT INTO travelcard.travel_account (member_id, name, balance) VALUES
(19, 'Alice', 1000),
(23, 'Bob', 2500),
(31, 'Charlie', 500),
(42, 'Diana', 7500)
ON CONFLICT (member_id) DO NOTHING;
44 changes: 44 additions & 0 deletions ps-cache-kotlin/pom.xml
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>3.4.4</version>
</parent>
<groupId>com.demo</groupId>
<artifactId>kotlin-app</artifactId>
<version>1.0.0</version>
<properties>
<java.version>21</java.version>
<kotlin.version>1.9.25</kotlin.version>
</properties>
<dependencies>
<dependency><groupId>org.springframework.boot</groupId><artifactId>spring-boot-starter-web</artifactId></dependency>
<dependency><groupId>org.springframework.boot</groupId><artifactId>spring-boot-starter-jdbc</artifactId></dependency>
<dependency><groupId>org.postgresql</groupId><artifactId>postgresql</artifactId></dependency>
<dependency><groupId>org.jetbrains.kotlin</groupId><artifactId>kotlin-reflect</artifactId></dependency>
<dependency><groupId>org.jetbrains.kotlin</groupId><artifactId>kotlin-stdlib</artifactId></dependency>
</dependencies>
<build>
<sourceDirectory>src/main/kotlin</sourceDirectory>
<plugins>
<plugin>
<groupId>org.jetbrains.kotlin</groupId>
<artifactId>kotlin-maven-plugin</artifactId>
<version>${kotlin.version}</version>
<configuration>
<compilerPlugins><plugin>spring</plugin></compilerPlugins>
<jvmTarget>${java.version}</jvmTarget>
</configuration>
<dependencies>
<dependency><groupId>org.jetbrains.kotlin</groupId><artifactId>kotlin-maven-allopen</artifactId><version>${kotlin.version}</version></dependency>
</dependencies>
<executions><execution><id>compile</id><goals><goal>compile</goal></goals></execution></executions>
</plugin>
<plugin><groupId>org.springframework.boot</groupId><artifactId>spring-boot-maven-plugin</artifactId></plugin>
</plugins>
</build>
</project>
89 changes: 89 additions & 0 deletions ps-cache-kotlin/src/main/kotlin/com/demo/App.kt
Original file line number Diff line number Diff line change
@@ -0,0 +1,89 @@
package com.demo

import org.springframework.boot.autoconfigure.SpringBootApplication
import org.springframework.boot.runApplication
import org.springframework.jdbc.core.JdbcTemplate
import org.springframework.web.bind.annotation.GetMapping
import org.springframework.web.bind.annotation.RequestParam
import org.springframework.web.bind.annotation.RestController
import org.springframework.http.ResponseEntity
import javax.sql.DataSource
import com.zaxxer.hikari.HikariDataSource

@SpringBootApplication
class App

fun main(args: Array<String>) {
runApplication<App>(*args)
}

data class Account(
val id: Int,
val memberId: Int,
val name: String,
val balance: Int
)

@RestController
class AccountController(private val jdbc: JdbcTemplate, private val dataSource: DataSource) {

@GetMapping("/health")
fun health() = mapOf("status" to "ok")

@GetMapping("/account")
fun getAccount(@RequestParam("member") memberId: Int): ResponseEntity<Any> {
val result = jdbc.execute(
org.springframework.jdbc.core.ConnectionCallback<Account?> { conn ->
conn.autoCommit = false
try {
conn.prepareStatement(
"""SELECT id, member_id, name, balance
FROM travelcard.travel_account
WHERE member_id = ?"""
).use { ps ->
ps.setInt(1, memberId)
ps.executeQuery().use { rs ->
val account = if (rs.next()) {
Account(
id = rs.getInt("id"),
memberId = rs.getInt("member_id"),
name = rs.getString("name"),
balance = rs.getInt("balance")
)
} else null

conn.commit()
account
Comment on lines +54 to +56
Copy link

Copilot AI Apr 16, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

conn.commit() is called while the ResultSet / PreparedStatement are still inside their use {} scopes. Some JDBC drivers can close cursors/result sets on commit, which can lead to surprising behavior. Consider moving the commit so it happens after the use blocks have completed (resources closed), and keeping rollback/cleanup centralized.

Copilot uses AI. Check for mistakes.
}
}
} catch (e: Exception) {
conn.rollback()
throw e
}
})

return if (result != null) {
ResponseEntity.ok(result)
} else {
ResponseEntity.status(404).body(mapOf("error" to "not found", "member_id" to memberId))
}
}

@GetMapping("/evict")
fun evict(): ResponseEntity<Map<String, Any>> {
Comment on lines +72 to +73
Copy link

Copilot AI Apr 16, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/evict changes server state (evicts pooled connections) but is exposed as a GET. For correctness with HTTP semantics (and to avoid accidental triggering by caches/prefetchers), consider making this a POST endpoint and updating test.sh accordingly.

Copilot uses AI. Check for mistakes.
val hikari = dataSource as? HikariDataSource
?: return ResponseEntity.status(500).body(mapOf("error" to "not a HikariDataSource"))

val mxBean = hikari.hikariPoolMXBean
?: return ResponseEntity.status(500).body(mapOf("error" to "pool MXBean not available"))

mxBean.softEvictConnections()
Thread.sleep(500)
Comment on lines +75 to +81
Copy link

Copilot AI Apr 16, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Avoid Thread.sleep on the request thread; it blocks the server thread pool and makes response latency non-deterministic. If you need to wait for eviction to take effect for the test, consider polling the pool state with a bounded timeout (or just return immediately and let the next /account request establish the new connection).

Suggested change
?: return ResponseEntity.status(500).body(mapOf("error" to "not a HikariDataSource"))
val mxBean = hikari.hikariPoolMXBean
?: return ResponseEntity.status(500).body(mapOf("error" to "pool MXBean not available"))
mxBean.softEvictConnections()
Thread.sleep(500)
?: return ResponseEntity.status(500).body(
mapOf(
"error" to "not a HikariDataSource",
"next_step" to "Verify that the application is configured to use HikariCP before calling /evict."
)
)
val mxBean = hikari.hikariPoolMXBean
?: return ResponseEntity.status(500).body(
mapOf(
"error" to "pool MXBean not available",
"next_step" to "Enable HikariCP pool management or confirm the datasource exposes a pool MXBean, then retry /evict."
)
)
mxBean.softEvictConnections()

Copilot uses AI. Check for mistakes.

return ResponseEntity.ok(mapOf(
"evicted" to true,
"active" to mxBean.activeConnections,
"idle" to mxBean.idleConnections
))
}
}
7 changes: 7 additions & 0 deletions ps-cache-kotlin/src/main/resources/application.properties
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
server.port=8080
spring.datasource.url=jdbc:postgresql://${DB_HOST:localhost}:${DB_PORT:5432}/${DB_NAME:testdb}?prepareThreshold=1&preparedStatementCacheQueries=256
Copy link

Copilot AI Apr 16, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The DB port defaults to 5432 in application.properties, but docker-compose.yml exposes Postgres on host port 5433. If the intended workflow includes running the app locally against the compose DB, consider aligning the default (DB_PORT) or the compose port mapping to avoid connection failures without extra env vars.

Suggested change
spring.datasource.url=jdbc:postgresql://${DB_HOST:localhost}:${DB_PORT:5432}/${DB_NAME:testdb}?prepareThreshold=1&preparedStatementCacheQueries=256
spring.datasource.url=jdbc:postgresql://${DB_HOST:localhost}:${DB_PORT:5433}/${DB_NAME:testdb}?prepareThreshold=1&preparedStatementCacheQueries=256

Copilot uses AI. Check for mistakes.
spring.datasource.username=${DB_USER:postgres}
spring.datasource.password=${DB_PASSWORD:postgres}
spring.datasource.hikari.maximum-pool-size=1
spring.datasource.hikari.minimum-idle=1
spring.sql.init.mode=never
38 changes: 38 additions & 0 deletions ps-cache-kotlin/test.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
#!/usr/bin/env bash
set -euo pipefail

BASE_URL="http://localhost:8080"

echo "=== PS-Cache Mock Mismatch Test (Kotlin/JDBC) ==="

echo "--- Window 1: Connection A ---"
echo " /account?member=19:"
curl -fSs "$BASE_URL/account?member=19"
echo ""
sleep 1

echo " /account?member=23:"
curl -fSs "$BASE_URL/account?member=23"
echo ""
sleep 1

echo ""
echo "--- Evict (force new connection) ---"
echo " /evict:"
curl -fSs "$BASE_URL/evict"
echo ""
sleep 1

echo ""
echo "--- Window 2: Connection B ---"
echo " /account?member=31:"
curl -fSs "$BASE_URL/account?member=31"
echo ""
sleep 1

echo " /account?member=42:"
curl -fSs "$BASE_URL/account?member=42"
echo ""

echo ""
echo "=== Done ==="
Loading