Skip to content

Commit

Permalink
🎉✨Add hosting of Snowflake ID tool
Browse files Browse the repository at this point in the history
⚡️Optimize core code

fixes #1, fixes #2
  • Loading branch information
LemonNoCry committed Dec 9, 2024
1 parent c7079a3 commit d76891d
Show file tree
Hide file tree
Showing 33 changed files with 893 additions and 116 deletions.
63 changes: 54 additions & 9 deletions README.en.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ registration, making it compatible with any framework or library using Snowflake
- **Flexible configuration**: Chainable API to customize registration logic.
- **High compatibility**: Supports .NET Standard 2.0, allowing cross-platform usage.
- **Simplifies development**: Reduces complexity in managing WorkerId for distributed systems.
- **High reliability**: Supports automatic renewal of WorkerId to prevent duplicate assignments.

---

Expand Down Expand Up @@ -173,22 +174,66 @@ long id = idGenInstance.NewLong();
Console.WriteLine($"Generated ID: {id}");
```

For other Snowflake ID libraries, follow a similar approach to pass the WorkerId obtained from
`SnowflakeId.AutoRegister`.
## AdvancedUsage

### Managing the Lifecycle of `Snowflake ID Tool Library`

Delegate the lifecycle of the Snowflake ID tool library to the `AutoRegister` instance to avoid the "zombie problem".
**Principle: Process A registers WorkerId 1, but due to various reasons (such as a short lifecycle, network issues, etc.), it cannot renew in time. In other processes, this
WorkerId is considered invalid, and process B will register the same WorkerId 1. When process A recovers, it will detect that WorkerId 1 is already in use and will cancel the
registration, re-registering the next time it is acquired.**

Usage only requires adjusting `Build`.

Here is an example of using `Yitter.IdGenerator`:

```csharp
//IAutoRegister => IAutoRegister<xxx>
static readonly IAutoRegister<IIdGenerator> AutoRegister = new AutoRegisterBuilder()

// Same as other configurations
...

// The key point is here
.Build<IIdGenerator>(config => new DefaultIdGenerator(new IdGeneratorOptions()
{
WorkerId = (ushort)config.WorkerId
}));

//Get Id
// Ensure to use `GetIdGenerator()` to get the `IdGenerator` instance each time, do not cache it, as it may re-register
long id =autoRegister.GetIdGenerator().NewLong();
Console.WriteLine($"Id: {id}");
```

### For other Snowflake ID generation libraries, refer to the above examples for integration.

---

## FAQ

- **What happens if the program crashes?**
WorkerId will not be released. On the next startup, the library attempts to reuse the previous WorkerId. If
unsuccessful, a new WorkerId is assigned.
* Q: Why do we need to auto-register WorkerId?
* A: Snowflake ID requires WorkerId to generate unique IDs. Auto-registering WorkerId can reduce the complexity of manual maintenance.


* Q: Will WorkerId be released if the program crashes?
* A: No. WorkerId has a lifecycle. If the program exits abnormally, it will try to register the previous WorkerId on the next startup. If it fails, it will re-register a new
WorkerId.


* Q: **What is the "zombie problem"?**
* A: **For example, process A registers a WorkerId, but due to various reasons (such as a short lifecycle, network issues, etc.), it cannot renew in time. In other processes, this
WorkerId is considered invalid, and process B will register the same WorkerId. If process A recovers, both process A and process B will use the same WorkerId, causing ID
duplication. See [Advanced Usage](#AdvancedUsage) for the solution.**


* Q: How to avoid multiple processes in the same file from being assigned the same WorkerId?
* A: Add a process-related identifier in SetExtraIdentifier, such as the current process ID.

- **How to prevent duplicate WorkerId across processes?**
Use `SetExtraIdentifier` with process-specific data, such as the current process ID.

- **Is the default storage mechanism suitable for production?**
No, it is recommended only for development and testing. Use Redis, SQL Server, or MySQL for production environments.
* Q: Is the default storage mechanism suitable for production environments?
* A: The default storage mechanism is only suitable for development and local testing (to maintain consistency). In production environments, it is recommended to use Redis, SQL
Server, MySQL, etc.

---

Expand Down
52 changes: 47 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@ SnowflakeId AutoRegister 是一个库,提供了一种简单的方法在 Snowfl
* 灵活配置:通过链式 API 自定义注册逻辑
* 高兼容性:支持 .NET Standard 2.0,可在多种平台运行
* 简化开发流程:减少手动维护 WorkerId 的复杂性
* 高可靠性:支持 WorkerId 的自动续期,避免重复分配

---

Expand Down Expand Up @@ -120,7 +121,7 @@ static readonly IAutoRegister AutoRegister = new AutoRegisterBuilder()
// 使用以下行使用 SQL Server 存储。
//.UseSqlServerStore("Server=localhost;Database=SnowflakeTest;User Id=sa;Password=123456;")
// Use the following line to use the MySQL store.
// 使用以下行使用 MySQL 存储。
.UseMySqlStore("Server=localhost;Port=3306;Database=snowflaketest;Uid=test;Pwd=123456;SslMode=None;")
.Build();
Expand Down Expand Up @@ -163,7 +164,7 @@ applicationLifetime.ApplicationStopping.Register(() =>

---

## 集成其他 Snowflake ID 库
## 集成 Snowflake ID 库

### Yitter.IdGenerator

Expand All @@ -180,22 +181,63 @@ long id = idGenInstance.NewLong();
Console.WriteLine($"Id: {id}");
```

## 高级用法

### 托管`雪花Id工具库`生命周期

将雪花Id工具库的生命周期托管到`AutoRegister`实例中,以避免`假死问题`
**原理:进程A注册了WorkerId 1,但是进程A因为各种原因(如生命周期太短、网络问题等)
导致无法及时续期,在其他进程看来此WorkerId已无效,进程B注册就会获得相同的WorkerId 1,在进程A恢复正常后,重新续期时会检测当前WorkId 1已被使用,会取消注册下次获取时会重新注册,**

用法只需要调整`Build`

一下是`Yitter.IdGenerator` 的用法示例:

```csharp
//IAutoRegister => IAutoRegister<xxx>
static readonly IAutoRegister<IIdGenerator> AutoRegister = new AutoRegisterBuilder()
//与其他配置一样
...
//重点在于这里
.Build<IIdGenerator>(config => new DefaultIdGenerator(new IdGeneratorOptions()
{
WorkerId = (ushort)config.WorkerId
}));
//获取Id
//确保每次都要使用`GetIdGenerator()`来获取`IdGenerator`实例,不要缓存,因为可能会重新注册
long id =autoRegister.GetIdGenerator().NewLong();
Console.WriteLine($"Id: {id}");
```
### 对于其他 Snowflake ID 生成库,可以参考上述示例进行集成。
---
## 常见问题 (FAQ)
* Q: 为什么需要自动注册 WorkerId?
* A: Snowflake ID 需要 WorkerId 来生成唯一的 ID。自动注册 WorkerId 可以减少手动维护的复杂性。
* Q: 如果程序崩溃了,WorkerId 会被释放吗?
* A: 不会。程序异常退出时,下次启动会尝试分配上一次的 WorkerId。如果失败,则重新注册新的 WorkerId。
* A: 不会。WorkerId存在生命周期,程序异常退出时,下次启动会尝试注册上一次的 WorkerId。如果失败,则重新注册新的 WorkerId。
* Q: **"假死问题"是什么?**
* A: **例如:进程A注册了WorkerId,但是进程A因为各种原因(如生命周期太短、网络问题等)
导致无法及时续期,在其他进程看来此WorkerId已无效,进程B注册就会获得相同的WorkerId,如果进程A恢复正常,此时进程A和进程B都会使用相同的WorkerId,导致ID重复
解决方案看[高级用法](#高级用法)**
* Q: 如何避免多进程重复分配 WorkerId?
* Q: 如何避免同文件多进程重复分配 WorkerId?
* A: 在 SetExtraIdentifier 中添加进程相关的标识符,例如当前进程 ID。
* Q: 默认存储机制适合生产环境吗?
* A: 默认存储机制仅适合开发和本地测试。在生产环境中,建议使用 Redis、SQL Server 或 MySQL 存储
* A: 默认存储机制仅适合开发和本地测试(为了保持一致性)。在生产环境中,建议使用 Redis、SQL Server、MySql等等
---
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

<PropertyGroup>
<PackageTags>Snowflake;SnowflakeId;AutoRegister;Id AutoRegister;</PackageTags>
<Version>1.0.1</Version>
<Version>1.0.2</Version>
<Description>SnowflakeId AutoRegister Db: A base library for SnowflakeId AutoRegister, providing database support for automatic WorkerId registration in SnowflakeId systems</Description>
</PropertyGroup>

Expand Down
20 changes: 9 additions & 11 deletions src/SnowflakeId.AutoRegister.Db/Storage/BaseDbStorage.cs
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ protected BaseDbStorage(BaseDbStorageOptions options, ISqlQueryProvider sqlQuery
ConnectionFactory = Options.ConnectionFactory ?? DefaultConnectionFactory;
}

#region disponse
#region Dispose

public void Dispose()
{
Expand Down Expand Up @@ -50,7 +50,7 @@ public bool Exist(string key)
public bool Set(string key, string value, int millisecond)
{
ClearExpiredValues();
return UseConnection((connection) =>
return UseTransaction((connection, transaction) =>
{
var parameters = new
{
Expand All @@ -59,15 +59,13 @@ public bool Set(string key, string value, int millisecond)
expireTime = DateTime.Now.AddMilliseconds(millisecond)
};

return connection.Execute(SqlQueryProvider.GetInsertOrUpdateQuery(Options.SchemaName), parameters) > 0;
return connection.Execute(SqlQueryProvider.GetInsertOrUpdateQuery(Options.SchemaName), parameters, transaction) > 0;
});
}

public bool SetNotExists(string key, string value, int millisecond)
{
ClearExpiredValues();

return UseConnection((connection) =>
return UseTransaction((connection, transaction) =>
{
var parameters = new
{
Expand All @@ -76,31 +74,31 @@ public bool SetNotExists(string key, string value, int millisecond)
expireTime = DateTime.Now.AddMilliseconds(millisecond)
};

return connection.Execute(SqlQueryProvider.GetInsertIfNotExistsQuery(Options.SchemaName), parameters) > 0;
return connection.Execute(SqlQueryProvider.GetInsertIfNotExistsQuery(Options.SchemaName), parameters, transaction) > 0;
});
}

public bool Expire(string key, int millisecond)
public bool Expire(string key, string value, int millisecond)
{
return UseConnection((connection) =>
{
var parameters = new
{
key,
key, value,
expireTime = DateTime.Now.AddMilliseconds(millisecond)
};

return connection.Execute(SqlQueryProvider.GetUpdateExpireQuery(Options.SchemaName), parameters) > 0;
});
}

public Task<bool> ExpireAsync(string key, int millisecond)
public Task<bool> ExpireAsync(string key, string value, int millisecond)
{
return UseConnectionAsync(Func);

async Task<bool> Func(DbConnection connection)
{
var parameters = new { key, expireTime = DateTime.Now.AddMilliseconds(millisecond) };
var parameters = new { key, value, expireTime = DateTime.Now.AddMilliseconds(millisecond) };

return await connection.ExecuteAsync(SqlQueryProvider.GetUpdateExpireQuery(Options.SchemaName), parameters) > 0;
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

<PropertyGroup>
<PackageTags>Snowflake;SnowflakeId;AutoRegister;Id AutoRegister;MySql</PackageTags>
<Version>1.0.2</Version>
<Version>1.0.3</Version>
<Description>SnowflakeId AutoRegister MySql: An extension of the SnowflakeId AutoRegister library, enabling automatic WorkerId registration in SnowflakeId systems using MySql</Description>
</PropertyGroup>

Expand Down
11 changes: 4 additions & 7 deletions src/SnowflakeId.AutoRegister.MySql/Storage/MySqlQueryProvider.cs
Original file line number Diff line number Diff line change
Expand Up @@ -17,16 +17,13 @@ public string GetInsertOrUpdateQuery(string schemaName) =>

public string GetInsertIfNotExistsQuery(string schemaName) =>
$"""
INSERT INTO `{schemaName}_RegisterKeyValues` (`Key`, `Value`, `ExpireTime`)
SELECT @key, @value, @expireTime
FROM DUAL
WHERE NOT EXISTS (
SELECT 1 FROM `{schemaName}_RegisterKeyValues` WHERE `Key` = @key
);
INSERT IGNORE INTO `{schemaName}_RegisterKeyValues` (`Key`, `Value`, `ExpireTime`)
VALUES (@key, @value, @expireTime);
""";


public string GetUpdateExpireQuery(string schemaName) =>
$"UPDATE `{schemaName}_RegisterKeyValues` SET `ExpireTime` = @expireTime WHERE `Key` = @key;";
$"UPDATE `{schemaName}_RegisterKeyValues` SET `ExpireTime` = @expireTime WHERE `Key` = @key AND `Value` = @value;";

public string GetDeleteQuery(string schemaName) =>
$"DELETE FROM `{schemaName}_RegisterKeyValues` WHERE `Key` = @key;";
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

<PropertyGroup>
<PackageTags>Snowflake;SnowflakeId;AutoRegister;Id AutoRegister;SqlServer</PackageTags>
<Version>1.0.4</Version>
<Version>1.0.5</Version>
<Description>SnowflakeId AutoRegister StackExchangeRedis: An extension of the SnowflakeId AutoRegister library, enabling automatic WorkerId registration in SnowflakeId systems using SqlServer
</Description>
</PropertyGroup>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ public string GetQuery(string schemaName) =>

public string GetInsertOrUpdateQuery(string schemaName) =>
$"""
MERGE INTO [{schemaName}].[RegisterKeyValues] AS Target
MERGE INTO [{schemaName}].[RegisterKeyValues] WITH (HOLDLOCK, UPDLOCK) AS Target
USING (VALUES (@key, @value, @expireTime)) AS Source ([Key], Value, ExpireTime)
ON Target.[Key] = Source.[Key]
WHEN MATCHED THEN
Expand All @@ -23,11 +23,16 @@ public string GetInsertIfNotExistsQuery(string schemaName) =>
$"""
INSERT INTO [{schemaName}].[RegisterKeyValues] ([Key], Value, ExpireTime)
SELECT @key, @value, @expireTime
WHERE NOT EXISTS (SELECT 1 FROM [{schemaName}].[RegisterKeyValues] WHERE [Key] = @key);
FROM (SELECT 1 AS tmp) AS t
WHERE NOT EXISTS (
SELECT 1
FROM [{schemaName}].[RegisterKeyValues] WITH (UPDLOCK, HOLDLOCK)
WHERE [Key] = @key
);
""";

public string GetUpdateExpireQuery(string schemaName) =>
$"UPDATE [{schemaName}].[RegisterKeyValues] SET ExpireTime = @expireTime WHERE [Key] = @key";
$"UPDATE [{schemaName}].[RegisterKeyValues] SET ExpireTime = @expireTime WHERE [Key] = @key AND [Value] = @value";

public string GetDeleteQuery(string schemaName) =>
$"DELETE FROM [{schemaName}].[RegisterKeyValues] WHERE [Key] = @key";
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

<PropertyGroup>
<PackageTags>Snowflake;SnowflakeId;AutoRegister;Id AutoRegister;StackExchange.Redis;Redis</PackageTags>
<Version>1.0.1</Version>
<Version>1.0.2</Version>
<Description>SnowflakeId AutoRegister StackExchangeRedis: An extension of the SnowflakeId AutoRegister library, enabling automatic WorkerId registration in SnowflakeId systems using StackExchange.Redis
</Description>
</PropertyGroup>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,16 @@
/// </summary>
internal class RedisStorage : IStorage
{
/// <summary>
/// Lua script: Checks if the key's value matches the provided value and sets the expiration time
/// </summary>
private const string ExpireScript = @"
if redis.call('get', KEYS[1]) == ARGV[1] then
return redis.call('pexpire', KEYS[1], ARGV[2])
else
return 0
end";

private readonly IConnectionMultiplexer _connection;
private readonly RedisStorageOption _storageOption;

Expand Down Expand Up @@ -46,16 +56,30 @@ public bool SetNotExists(string key, string value, int millisecond)
return db.StringSet(_storageOption.InstanceName + key, value, TimeSpan.FromMilliseconds(millisecond), When.NotExists);
}

public bool Expire(string key, int millisecond)
public bool Expire(string key, string value, int millisecond)
{
var db = _connection.GetDatabase();
return db.KeyExpire(_storageOption.InstanceName + key, TimeSpan.FromMilliseconds(millisecond));
var redisKey = _storageOption.InstanceName + key;

var result = (int)db.ScriptEvaluate(
ExpireScript,
[redisKey],
[value, millisecond]);

return result == 1;
}

public Task<bool> ExpireAsync(string key, int millisecond)
public async Task<bool> ExpireAsync(string key, string value, int millisecond)
{
var db = _connection.GetDatabase();
return db.KeyExpireAsync(_storageOption.InstanceName + key, TimeSpan.FromMilliseconds(millisecond));
var redisKey = _storageOption.InstanceName + key;

var result = (int)await db.ScriptEvaluateAsync(
ExpireScript,
[redisKey],
[value, millisecond]);

return result == 1;
}

public bool Delete(string key)
Expand All @@ -65,7 +89,7 @@ public bool Delete(string key)
}


#region disponse
#region Dispose

public void Dispose()
{
Expand Down
Loading

0 comments on commit d76891d

Please sign in to comment.