Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a Balancer interface type which can proxy data between io.ReadWriteCloser types #2

Open
TheApeMachine opened this issue Nov 24, 2022 · 0 comments
Assignees

Comments

@TheApeMachine
Copy link
Owner

Assuming most objects at least implement the io.ReadWriteCloser interface
And such objects can be either duplicated, distributed, or act as performance helpers
A Balancer object can be seen as an abstraction to implement a Strategy to select which objects are selected to proxy a Workload to.

USE CASE:

Retrieving data from a Data Lake built on S3 compatible storage is fast
Retrieving data from an in-memory data store (essentially a cache) is even faster
Both stores are using a Prefix based approach to provide key/value storage (they are 1:1 compatible)
An object implementing the Balancer interface would be passed a Strategy called FirstToFinish (or something like that)
FirstToFinish calls both stores, the S3 compatible and the in-memory store
Which ever store returns the data first will cancel out the request to the other one
This will always provide the quickest path to results, while also having a fallback to the slower path

PRACTICAL ABSTRACT EXAMPLE:

package main

/*
S3Store is a client to a data lake built on AWS S3 (or compatible) storage.
*/
type S3Store struct {
    io.ReadWriteCloser
}

// ... Read, Write, Close methods omitted.

/*
RadixStore is an in-memory Radix Trie.
*/
type RadixStore struct {
    io.ReadWriteCloser
}

// ... Read, Write, Close methods omitted.

/*
Balancer is an interface to implement if an object should be able to balance a Workload to two or more objects implementing the io.ReadWriteCloser interface. Itself it also implements the io.ReadWriteCloser to make the usage of the interface type transparent to the direct usage of the Store types.
*/
type Balancer interface {
    io.ReadWriteCloser
}

/*
NewBalancer takes in a compatible struct type as a balancer Strategy and converts it to a Balancer interface type.
*/
func NewBalancer(balancerStrategy Balancer) Balancer {
    return balancerStrategy
}

/*
ProtoBalancer serves as the most common implementation of a Strategy acting as a Balancer type.
It just uses a random object from the pool.
*/
type ProtoBalancer struct {
    pool       []io.ReadWriteCloser
    selector *rand.Rand
}

/*
NewProtoBalancer constructs a Balancer Strategy that uses random pool selection.
*/
func NewProtoBalancer(pool []io.ReadWriteCloser) ProtoBalancer {
    return ProtoBalancer{
        pool:        pool,
        selector: rand.New(rand.NewSource(time.Now().Unix())),
    }
}

/*
Close the objects in the Balancer pool and cleanup.
*/
func (proto ProtoBalancer) Close() error {
    for _, p := range proto.pool {
        err := p.Close()
        errnie.Handles(err)
    }
}

/*
Read selects a random object from the pool and proxies the workload.
*/
func (proto ProtoBalancer) Read(p []byte) (n int, err error) {
    return proto.pool[proto.selector.Intn(len(proto.pool))].Read(p)
} 

/*
Write selects a random object from the pool and proxies the workload.
*/
func (proto ProtoBalancer) Write(p []byte) (n int, err error) {
    return proto.pool[proto.selector.Intn(len(proto.pool))].Write(p)
}

func main() {
    // Instantiate a new balancer type.
    balancer := NewBalancer(NewProtoBalancer(S3Store{}, RadixStore{}))

    // Make some test data.
    buf := []byte("some test data")

    // User io.Copy(dst, src) to copy the data to the balancer, which will take care of the rest.
    io.Copy(balancer, buf)
}
@TheApeMachine TheApeMachine self-assigned this Nov 24, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant