Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Data Race in Coordinator #956

Closed
hongyunyan opened this issue Jan 26, 2025 · 0 comments · Fixed by #957
Closed

Data Race in Coordinator #956

hongyunyan opened this issue Jan 26, 2025 · 0 comments · Fixed by #957
Assignees
@hongyunyan
Labels
type/bug The issue is confirmed as a bug.

Comments

@hongyunyan
Copy link
Collaborator

What did you do?

run the available integration test

What did you expect to see?

pass

What did you see instead?

failed because data race

=== Command to ticdc(new arch).
==================
WARNING: DATA RACE
Read at 0x00c003c18690 by goroutine 1275:
  github.com/pingcap/ticdc/coordinator.(*Controller).Stop()
      /home/runner/work/ticdc/ticdc/coordinator/controller.go:397 +0x16b
  github.com/pingcap/ticdc/coordinator.(*coordinator).AsyncStop()
      /home/runner/work/ticdc/ticdc/coordinator/coordinator.go:354 +0xe4
  github.com/pingcap/ticdc/api/v2.(*OpenAPIV2).resignOwner()
      /home/runner/work/ticdc/ticdc/api/v2/coordinator.go:34 +0xc6
  github.com/pingcap/ticdc/api/v2.(*OpenAPIV2).resignOwner-fm()
      <autogenerated>:1 +0x3d
  github.com/gin-gonic/gin.(*Context).Next()
      /home/runner/go/pkg/mod/github.com/gin-gonic/[email protected]/context.go:174 +0x92
  github.com/pingcap/ticdc/api/v2.RegisterOpenAPIV2Routes.ForwardToCoordinatorMiddleware.func4()
      /home/runner/work/ticdc/ticdc/api/middleware/middleware.go:110 +0x73
  github.com/gin-gonic/gin.(*Context).Next()
      /home/runner/go/pkg/mod/github.com/gin-gonic/[email protected]/context.go:174 +0x92
  github.com/pingcap/ticdc/api/v2.RegisterOpenAPIV2Routes.ErrorHandleMiddleware.func2()
      /home/runner/work/ticdc/ticdc/api/middleware/middleware.go:49 +0x44
  github.com/gin-gonic/gin.(*Context).Next()
      /home/runner/go/pkg/mod/github.com/gin-gonic/[email protected]/context.go:174 +0x92
  github.com/pingcap/ticdc/api/v2.RegisterOpenAPIV2Routes.LogMiddleware.func1()
      /home/runner/work/ticdc/ticdc/api/middleware/middleware.go:74 +0x19b
  github.com/gin-gonic/gin.(*Context).Next()
      /home/runner/go/pkg/mod/github.com/gin-gonic/[email protected]/context.go:174 +0x15a
  github.com/gin-gonic/gin.CustomRecoveryWithWriter.func1()
      /home/runner/go/pkg/mod/github.com/gin-gonic/[email protected]/recovery.go:102 +0xd4
  github.com/gin-gonic/gin.(*Context).Next()
      /home/runner/go/pkg/mod/github.com/gin-gonic/[email protected]/context.go:174 +0xbb5
  github.com/gin-gonic/gin.(*Engine).handleHTTPRequest()
      /home/runner/go/pkg/mod/github.com/gin-gonic/[email protected]/gin.go:620 +0x7b4
  github.com/gin-gonic/gin.(*Engine).ServeHTTP()
      /home/runner/go/pkg/mod/github.com/gin-gonic/[email protected]/gin.go:576 +0x418
  net/http.serverHandler.ServeHTTP()
      /opt/hostedtoolcache/go/1.23.4/x64/src/net/http/server.go:3210 +0x2a1
  net/http.(*conn).serve()
      /opt/hostedtoolcache/go/1.23.4/x64/src/net/http/server.go:2092 +0x12a4
  net/http.(*Server).Serve.gowrap3()
      /opt/hostedtoolcache/go/1.23.4/x64/src/net/http/server.go:3360 +0x4f

Previous write at 0x00c003c18690 by goroutine 1138:
  github.com/pingcap/ticdc/coordinator.(*Controller).FinishBootstrap()
      /home/runner/work/ticdc/ticdc/coordinator/controller.go:392 +0x1c38
  github.com/pingcap/ticdc/coordinator.(*Controller).onBootstrapDone()
      /home/runner/work/ticdc/ticdc/coordinator/controller.go:267 +0x53a
  github.com/pingcap/ticdc/coordinator.(*Controller).onNodeChanged()
      /home/runner/work/ticdc/ticdc/coordinator/controller.go:218 +0xa59
  github.com/pingcap/ticdc/coordinator.(*Controller).checkOnNodeChanged()
      /home/runner/work/ticdc/ticdc/coordinator/controller.go:168 +0x118
  github.com/pingcap/ticdc/coordinator.(*Controller).HandleEvent()
      /home/runner/work/ticdc/ticdc/coordinator/controller.go:153 +0x171
  github.com/pingcap/ticdc/coordinator.(*coordinator).runHandleEvent()
      /home/runner/work/ticdc/ticdc/coordinator/coordinator.go:195 +0xda
  github.com/pingcap/ticdc/coordinator.(*coordinator).Run.gowrap3()
      /home/runner/work/ticdc/ticdc/coordinator/coordinator.go:159 +0x4f

Goroutine 1275 (running) created at:
  net/http.(*Server).Serve()
      /opt/hostedtoolcache/go/1.23.4/x64/src/net/http/server.go:3360 +0x8ec
  github.com/pingcap/ticdc/server.(*HttpServer).Run()
      /home/runner/work/ticdc/ticdc/server/module_http.go:77 +0x1f7
  github.com/pingcap/ticdc/server.(*server).Run.(*server).Run.func2.func3()
      /home/runner/work/ticdc/ticdc/server/server.go:237 +0x36e
  golang.org/x/sync/errgroup.(*Group).Go.func1()
      /home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:78 +0x91

Goroutine 1138 (running) created at:
  github.com/pingcap/ticdc/coordinator.(*coordinator).Run()
      /home/runner/work/ticdc/ticdc/coordinator/coordinator.go:159 +0x444
  github.com/pingcap/ticdc/server.(*elector).campaignCoordinator()
      /home/runner/work/ticdc/ticdc/server/module_election.go:139 +0xef6
  github.com/pingcap/ticdc/server.(*elector).Run.func1()
      /home/runner/work/ticdc/ticdc/server/module_election.go:58 +0x58
  golang.org/x/sync/errgroup.(*Group).Go.func1()
      /home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:78 +0x91
==================
==================
WARNING: DATA RACE
Read at 0x00c003782b88 by goroutine 1275:
  github.com/pingcap/ticdc/coordinator.(*Controller).Stop()
      /home/runner/work/ticdc/ticdc/coordinator/controller.go:397 +0x1a5
  github.com/pingcap/ticdc/coordinator.(*coordinator).AsyncStop()
      /home/runner/work/ticdc/ticdc/coordinator/coordinator.go:354 +0xe4
  github.com/pingcap/ticdc/api/v2.(*OpenAPIV2).resignOwner()
      /home/runner/work/ticdc/ticdc/api/v2/coordinator.go:34 +0xc6
  github.com/pingcap/ticdc/api/v2.(*OpenAPIV2).resignOwner-fm()
      <autogenerated>:1 +0x3d
  github.com/gin-gonic/gin.(*Context).Next()
      /home/runner/go/pkg/mod/github.com/gin-gonic/[email protected]/context.go:174 +0x92
  github.com/pingcap/ticdc/api/v2.RegisterOpenAPIV2Routes.ForwardToCoordinatorMiddleware.func4()
      /home/runner/work/ticdc/ticdc/api/middleware/middleware.go:110 +0x73
  github.com/gin-gonic/gin.(*Context).Next()
      /home/runner/go/pkg/mod/github.com/gin-gonic/[email protected]/context.go:174 +0x92
  github.com/pingcap/ticdc/api/v2.RegisterOpenAPIV2Routes.ErrorHandleMiddleware.func2()
      /home/runner/work/ticdc/ticdc/api/middleware/middleware.go:49 +0x44
  github.com/gin-gonic/gin.(*Context).Next()
      /home/runner/go/pkg/mod/github.com/gin-gonic/[email protected]/context.go:174 +0x92
  github.com/pingcap/ticdc/api/v2.RegisterOpenAPIV2Routes.LogMiddleware.func1()
      /home/runner/work/ticdc/ticdc/api/middleware/middleware.go:74 +0x19b
  github.com/gin-gonic/gin.(*Context).Next()
      /home/runner/go/pkg/mod/github.com/gin-gonic/[email protected]/context.go:174 +0x15a
  github.com/gin-gonic/gin.CustomRecoveryWithWriter.func1()
      /home/runner/go/pkg/mod/github.com/gin-gonic/[email protected]/recovery.go:102 +0xd4
  github.com/gin-gonic/gin.(*Context).Next()
      /home/runner/go/pkg/mod/github.com/gin-gonic/[email protected]/context.go:174 +0xbb5
  github.com/gin-gonic/gin.(*Engine).handleHTTPRequest()
      /home/runner/go/pkg/mod/github.com/gin-gonic/[email protected]/gin.go:620 +0x7b4
  github.com/gin-gonic/gin.(*Engine).ServeHTTP()
      /home/runner/go/pkg/mod/github.com/gin-gonic/[email protected]/gin.go:576 +0x418
  net/http.serverHandler.ServeHTTP()
      /opt/hostedtoolcache/go/1.23.4/x64/src/net/http/server.go:3210 +0x2a1
  net/http.(*conn).serve()
      /opt/hostedtoolcache/go/1.23.4/x64/src/net/http/server.go:2092 +0x12a4
  net/http.(*Server).Serve.gowrap3()
      /opt/hostedtoolcache/go/1.23.4/x64/src/net/http/server.go:3360 +0x4f

Previous write at 0x00c003782b88 by goroutine 1138:
  github.com/pingcap/ticdc/coordinator.(*Controller).FinishBootstrap()
      /home/runner/work/ticdc/ticdc/coordinator/controller.go:392 +0x1bd6
  github.com/pingcap/ticdc/coordinator.(*Controller).onBootstrapDone()
      /home/runner/work/ticdc/ticdc/coordinator/controller.go:267 +0x53a
  github.com/pingcap/ticdc/coordinator.(*Controller).onNodeChanged()
      /home/runner/work/ticdc/ticdc/coordinator/controller.go:218 +0xa59
  github.com/pingcap/ticdc/coordinator.(*Controller).checkOnNodeChanged()
      /home/runner/work/ticdc/ticdc/coordinator/controller.go:168 +0x118
  github.com/pingcap/ticdc/coordinator.(*Controller).HandleEvent()
      /home/runner/work/ticdc/ticdc/coordinator/controller.go:153 +0x171
  github.com/pingcap/ticdc/coordinator.(*coordinator).runHandleEvent()
      /home/runner/work/ticdc/ticdc/coordinator/coordinator.go:195 +0xda
  github.com/pingcap/ticdc/coordinator.(*coordinator).Run.gowrap3()
      /home/runner/work/ticdc/ticdc/coordinator/coordinator.go:159 +0x4f

Goroutine 1275 (running) created at:
  net/http.(*Server).Serve()
      /opt/hostedtoolcache/go/1.23.4/x64/src/net/http/server.go:3360 +0x8ec
  github.com/pingcap/ticdc/server.(*HttpServer).Run()
      /home/runner/work/ticdc/ticdc/server/module_http.go:77 +0x1f7
  github.com/pingcap/ticdc/server.(*server).Run.(*server).Run.func2.func3()
      /home/runner/work/ticdc/ticdc/server/server.go:237 +0x36e
  golang.org/x/sync/errgroup.(*Group).Go.func1()
      /home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:78 +0x91

Goroutine 1138 (running) created at:
  github.com/pingcap/ticdc/coordinator.(*coordinator).Run()
      /home/runner/work/ticdc/ticdc/coordinator/coordinator.go:159 +0x444
  github.com/pingcap/ticdc/server.(*elector).campaignCoordinator()
      /home/runner/work/ticdc/ticdc/server/module_election.go:139 +0xef6
  github.com/pingcap/ticdc/server.(*elector).Run.func1()
      /home/runner/work/ticdc/ticdc/server/module_election.go:58 +0x58
  golang.org/x/sync/errgroup.(*Group).Go.func1()
      /home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:78 +0x91
==================
==================
WARNING: DATA RACE
Read at 0x00c006affa10 by goroutine 1275:
  github.com/pingcap/ticdc/utils/threadpool.(*TaskHandle).Cancel()
      /home/runner/work/ticdc/ticdc/utils/threadpool/task.go:55 +0x231
  github.com/pingcap/ticdc/coordinator.(*Controller).Stop()
      /home/runner/work/ticdc/ticdc/coordinator/controller.go:398 +0x1d7
  github.com/pingcap/ticdc/coordinator.(*coordinator).AsyncStop()
      /home/runner/work/ticdc/ticdc/coordinator/coordinator.go:354 +0xe4
  github.com/pingcap/ticdc/api/v2.(*OpenAPIV2).resignOwner()
      /home/runner/work/ticdc/ticdc/api/v2/coordinator.go:34 +0xc6
  github.com/pingcap/ticdc/api/v2.(*OpenAPIV2).resignOwner-fm()
      <autogenerated>:1 +0x3d
  github.com/gin-gonic/gin.(*Context).Next()
      /home/runner/go/pkg/mod/github.com/gin-gonic/[email protected]/context.go:174 +0x92
  github.com/pingcap/ticdc/api/v2.RegisterOpenAPIV2Routes.ForwardToCoordinatorMiddleware.func4()
      /home/runner/work/ticdc/ticdc/api/middleware/middleware.go:110 +0x73
  github.com/gin-gonic/gin.(*Context).Next()
      /home/runner/go/pkg/mod/github.com/gin-gonic/[email protected]/context.go:174 +0x92
  github.com/pingcap/ticdc/api/v2.RegisterOpenAPIV2Routes.ErrorHandleMiddleware.func2()
      /home/runner/work/ticdc/ticdc/api/middleware/middleware.go:49 +0x44
  github.com/gin-gonic/gin.(*Context).Next()
      /home/runner/go/pkg/mod/github.com/gin-gonic/[email protected]/context.go:174 +0x92
  github.com/pingcap/ticdc/api/v2.RegisterOpenAPIV2Routes.LogMiddleware.func1()
      /home/runner/work/ticdc/ticdc/api/middleware/middleware.go:74 +0x19b
  github.com/gin-gonic/gin.(*Context).Next()
      /home/runner/go/pkg/mod/github.com/gin-gonic/[email protected]/context.go:174 +0x15a
  github.com/gin-gonic/gin.CustomRecoveryWithWriter.func1()
      /home/runner/go/pkg/mod/github.com/gin-gonic/[email protected]/recovery.go:102 +0xd4
  github.com/gin-gonic/gin.(*Context).Next()
      /home/runner/go/pkg/mod/github.com/gin-gonic/[email protected]/context.go:174 +0xbb5
  github.com/gin-gonic/gin.(*Engine).handleHTTPRequest()
      /home/runner/go/pkg/mod/github.com/gin-gonic/[email protected]/gin.go:620 +0x7b4
  github.com/gin-gonic/gin.(*Engine).ServeHTTP()
      /home/runner/go/pkg/mod/github.com/gin-gonic/[email protected]/gin.go:576 +0x418
  net/http.serverHandler.ServeHTTP()
      /opt/hostedtoolcache/go/1.23.4/x64/src/net/http/server.go:3210 +0x2a1
  net/http.(*conn).serve()
      /opt/hostedtoolcache/go/1.23.4/x64/src/net/http/server.go:2092 +0x12a4
  net/http.(*Server).Serve.gowrap3()
      /opt/hostedtoolcache/go/1.23.4/x64/src/net/http/server.go:3360 +0x4f

Previous write at 0x00c006affa10 by goroutine 1138:
  github.com/pingcap/ticdc/utils/threadpool.(*threadPoolImpl).Submit()
      /home/runner/work/ticdc/ticdc/utils/threadpool/thread_pool.go:55 +0x19a
  github.com/pingcap/ticdc/coordinator.(*Controller).FinishBootstrap()
      /home/runner/work/ticdc/ticdc/coordinator/controller.go:391 +0x1b49
  github.com/pingcap/ticdc/coordinator.(*Controller).onBootstrapDone()
      /home/runner/work/ticdc/ticdc/coordinator/controller.go:267 +0x53a
  github.com/pingcap/ticdc/coordinator.(*Controller).onNodeChanged()
      /home/runner/work/ticdc/ticdc/coordinator/controller.go:218 +0xa59
  github.com/pingcap/ticdc/coordinator.(*Controller).checkOnNodeChanged()
      /home/runner/work/ticdc/ticdc/coordinator/controller.go:168 +0x118
  github.com/pingcap/ticdc/coordinator.(*Controller).HandleEvent()
      /home/runner/work/ticdc/ticdc/coordinator/controller.go:153 +0x171
  github.com/pingcap/ticdc/coordinator.(*coordinator).runHandleEvent()
      /home/runner/work/ticdc/ticdc/coordinator/coordinator.go:195 +0xda
  github.com/pingcap/ticdc/coordinator.(*coordinator).Run.gowrap3()
      /home/runner/work/ticdc/ticdc/coordinator/coordinator.go:159 +0x4f

Goroutine 1275 (running) created at:
  net/http.(*Server).Serve()
      /opt/hostedtoolcache/go/1.23.4/x64/src/net/http/server.go:3360 +0x8ec
  github.com/pingcap/ticdc/server.(*HttpServer).Run()
      /home/runner/work/ticdc/ticdc/server/module_http.go:77 +0x1f7
  github.com/pingcap/ticdc/server.(*server).Run.(*server).Run.func2.func3()
      /home/runner/work/ticdc/ticdc/server/server.go:237 +0x36e
  golang.org/x/sync/errgroup.(*Group).Go.func1()
      /home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:78 +0x91

Goroutine 1138 (running) created at:
  github.com/pingcap/ticdc/coordinator.(*coordinator).Run()
      /home/runner/work/ticdc/ticdc/coordinator/coordinator.go:159 +0x444
  github.com/pingcap/ticdc/server.(*elector).campaignCoordinator()
      /home/runner/work/ticdc/ticdc/server/module_election.go:139 +0xef6
  github.com/pingcap/ticdc/server.(*elector).Run.func1()
      /home/runner/work/ticdc/ticdc/server/module_election.go:58 +0x58
  golang.org/x/sync/errgroup.(*Group).Go.func1()
      /home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:78 +0x91
==================

Versions of the cluster

Upstream TiDB cluster version (execute SELECT tidb_version(); in a MySQL client):

(paste TiDB cluster version here)

Upstream TiKV version (execute tikv-server --version):

(paste TiKV version here)

TiCDC version (execute cdc version):

(paste TiCDC version here)
@hongyunyan hongyunyan added the type/bug The issue is confirmed as a bug. label Jan 26, 2025
@hongyunyan hongyunyan self-assigned this Jan 26, 2025
@hongyunyan hongyunyan mentioned this issue Jan 26, 2025
Merged
@ti-chi-bot ti-chi-bot bot closed this as completed in #957 Jan 27, 2025
@ti-chi-bot ti-chi-bot bot closed this as completed in 38ec3a4 Jan 27, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@hongyunyan hongyunyan

Labels
type/bug The issue is confirmed as a bug.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix data race in coordinator hongyunyan/tigate
1 participant
@hongyunyan