Johan Forsberg, johan.forsberg@maxiv.lu.se
Taranta workshop 2023

"TangoGQL Brain storming"


# Background

GraphQL schema allowing custom queries against Tango control system

TangoGQL provides all Tango communication for Taranta, including "live" subscriptions


# Current situation

-   Code has grown a bit messy

-   Some dependency versions are stale (graphene, graphql-ws), likely requiring refactoring

-   Queries are inefficient


# Call multiplication

A common issue with GraphQL schema that they don't fit 100% with the underlying data model. This is very true for TangoGQL.

Currently TangoGQL talks to Tango in most resolvers. This leads to multiplying calls, the more "nested" the query is.

E.g. if we query 10 devices, and read 10 properties on each, this will lead to 100 db calls, each reading one property.

This means it's easy to write a simple query that becomes unexpectedly slow; especially with wildcards.


# Data loaders

**Data loaders** is a clever way to combine calls and reuse results from the backend during a single query.

So if we e.g. read properties from several devices in the same query, we can combine them into fewer, or even **one** Tango DB call (may require using the "infamous" `DbMySqlSelect` command ;).

Data loaders also cache results so that if they are repeated **in the same query**, they are reused.

The end result is faster response and lower load on the control system.


# Ariadne

-   <https://github.com/mirumee/ariadne>

-   Fairly new (but not too new) GraphQL Python framework

-   Reads a GraphQL schema file and you implement resolvers for all the types

-   Seems simple and flexible, low on "boilerplate"

-   Backend built on Starlette (same as FastAPI)

-   Support for data loaders


# idea: tangogql-ariadne

TangoGQL implementation using Ariadne

-   Schema mostly complete (partially tested with Taranta)

-   Test suite adapted from tangogql

-   Missing some Taranta specific stuff, like auth (is this specified somewhere?)

-   Uses data loaders

-   Should benchmark!

-   Should publish!


# Subscription performance

How do we "scale up" TangoGQL subscriptions?

"Naive" load balancing with multiple instances will result in a lot of the same subscriptions on several instances, increasing control system traffic

Load balancing websockets based on query data..?


# Idea: "Broker" pattern with Redis

                                                        /---->Device
    Taranta ----> TangoGQL---\                /---Worker----->Device
                              +--> Broker <--+----Worker  
                  TangoGQL---/                \---Worker

-   Tango attribute subscriptions moved into scalable number of "workers"

-   Each worker handles attributes for a subset of devices

-   Workers agree on which devices to handle using a "hash ring" on Redis

-   Workers publish events on Redis "pub-sub" channels

-   TangoGQL subscribes to pub-sub channels for the attributes they need

-   Multiple TangoGQL instances no problem, they cause no extra control system load

Prototype exists, needs testing on a larger scale

Feedback:

-   High complexity for unsure gains&#x2026; need a strong motivation
-   Reimplementing subscriptions in C++ would perhaps be a better optimization
    (Wouldn't need to cover the whole TangoGQL API as it's a separate endpoint)
    Though, most of the machinery is already C++, it's just the setup of the websocket
    and some message handling that happens in python, not sure how much we'd gain.


# Broker drawbacks

-   Single point of failure

-   Bottleneck (though it is likely to be much faster than TangoGQL!)

-   More complex setup (should be optional)

-   Alternatives to Redis?
    -   Dragonfly is a "drop in" that claims higher performance
    
    -   RabbitMQ?


# Multiple vs single GraphQL instances


## One instance (at least) per control system

-   This is what we have today
-   Authentication is on this level (i.e. can you change this Tango control system)
-   Load balancing

-   ?

## One global instance

-   Some network benefit?

-   Bottleneck for all communication
-   Need to implement communication with multiple DBs
-   Authentication needs to be smarter (what about single query to multiple db?)

    With multi-db support, attributes will include tango host, e.g. my-host:10000/a/b/c/hello


# Idea: GraphQL schema V2?

I think the schema is in need of a minor cleanup, mostly naming.

-   Casing (`writeValue` but `minvalue`?) `min_value`

-   Confusion (`server: DeviceInfo`???)

-   Inconsistency with Tango API (`setAttributeValue` vs `write_attribute`)

Maybe there are some missing things we should add?

-   config events?
-   interface events?
-   add "events" or "polling" to event data, to tell how the data was acquired

/db
/v2/db


# Thanks!

Questions?