Overview

This, essentially, involves the creation of a new endpoint, similar to the /skyblock/auctions endpoint, listing the most recently updated players on the network.

This would be updated on a frequent basis by a singular handler function, and would be served directly from Cloudflare.

Implementation Cost

This is an extremely cheap and quick implementation. The listener would simply listen to a MongoDB Change Stream for when a player’s data updates, and add it to a cache to be served.

There are two ways to implement this solution, each with varying implications, so I will break these categories down by each implementation.

Type: Full Data

This would mean specifying the fullDocument: updateLookup option when initialising the change stream. Essentially, this tells the driver to return the full player document when a change is detected, rather than just the field delta.

When serving on the API, this means that each document will contain all of the player data, so the filtering can use the exact same processing as the existing endpoints before serving the document.

Example implementation:

ConcurrentLinkedQueue<Player> playersUpdatedThisMinute = new ConcurrentLinkedQueue();
List<Player> playersUpdatedLastMinute = new ArrayList();
long lastMinutePushTime = 0;

public void startChangeStreamListener() {
		ChangeStreamIterable<Document> changeStream = playerCollection.watch(List.of(
        new Document("$match", new Document("operationType", "update"))
    )).fullDocument(FullDocument.UPDATE_LOOKUP);
		
		for (ChangeStreamDocument<Document> document : changeStream) {
        playersThisMinute.add(Player.of(document.getFullDocument()));
    }
}

public synchronized List<Player> getPlayersUpdatedLastMinute() {
    if (System.currentTimeMillis() - lastMinutePushTime < 60_000) {
        // Only update once a minute.
        return playersUpdatedLastMinute;
    }
    lastMinutePushTime = System.currentTimeMillis();
    // Define a new ArrayList for all the players updated in the last minute.
    int playerCount = playersUpdatedThisMinute.size();
    playersUpdatedLastMinute = new ArrayList(playerCount);

    for (int i = 0; i < playerCount; i++) {
        Player player = playersUpdatedThisMinute.poll();
        // Player should never be null unless you're accessing this concurrently.
        if (player != null) {
            playersUpdatedLastMinute.add(player);
        }
    }
    return playersUpdatedLastMinute;
}

As you can tell, this is a very simple implementation. Code for generating the pages has been omitted, but would be similar to the existing /skyblock/auctions endpoint.

Type: Partial Data

This is the default behaviour of a MongoDB change stream. The ChangeStreamDocument only includes the fields that were updated in the update - this is the least intensive for the database, but may require additional behaviour, depending on how the existing Hypixel API infrastructure handles excluding private fields.

Assumedly, however, it is a simple MongoDB $project specification, and so the implementation would be similar to this:

ConcurrentLinkedQueue<JsonObject> playersUpdatedThisMinute = new ConcurrentLinkedQueue();
List<JsonObject> playersUpdatedLastMinute = new ArrayList();
long lastMinutePushTime = 0;

public void startChangeStreamListener() {
		ChangeStreamIterable<Document> changeStream = playerCollection.watch(List.of(
        new Document("$match", new Document("operationType", "update"))
        new Document("$project")
    ));
		
		for (ChangeStreamDocument<Document> document : changeStream) {
        JsonObject updatedData = new JsonObject();
        for (Map.Entry<String, BsonValue> entry : document.getUpdateDescription()
                .getUpdatedFields().entrySet()) {
            // Value would require processing here into a valid JSON value. 
            updatedData.add(entry.getKey(), entry.getValue());
        }
    }
}

public synchronized List<JsonObject> getPlayersUpdatedLastMinute() {
    if (System.currentTimeMillis() - lastMinutePushTime < 60_000) {
        // Only update once a minute.
        return playersUpdatedLastMinute;
    }
    lastMinutePushTime = System.currentTimeMillis();
    // Define a new ArrayList for all the players updated in the last minute.
    int playerCount = playersUpdatedThisMinute.size();
    playersUpdatedLastMinute = new ArrayList(playerCount);

    for (int i = 0; i < playerCount; i++) {
        JsonObject player = playersUpdatedThisMinute.poll();
        // Player should never be null unless you're accessing this concurrently.
        if (player != null) {
            playersUpdatedLastMinute.add(player);
        }
    }
    return playersUpdatedLastMinute;
}

This is a very similar implementation to the full data implementation, except the update data would have to be parsed to individual json update data rather than identical player information to the other endpoints. This would entail more work for both Hypixel and the end-user, so it may not be the best solution.

However, overall the implementation cost is very low.

Maintenance Cost

Change Streams are remarkably light operations. They operate based on the oplog, which is the same collection that replica sets use when replicating data from the primary. This means that load on the players database and collection is extremely low, and most of the data is cached as it’s needed to write the operation data to the replicas.