Questions about the pipeline

[hxan]

Hello, I read the code of this project. There are some points in the Gpipe related code that I don’t understand, such as what the number of GPUs in each pipeline means. My understanding is that each pipeline has only one GPU running.

[hxan]

There are still things wrong with the program, like when using nccl as a backend, the nccl_backend.py doesn't implement the isend operation, and it will report an error when it runs