Make pytest-incremental assume a full run has already been executed

[nicoddemus]

Hi,

First of all, thanks for the work on the plugin! 😁

I would like to request a new feature/workflow.

Let me describe my use case:

Before I start working on a new feature, I checkout a "known good" version of the repository, where all tests have passed on our CI servers. I would love to be able to start working by changing some files and have pytest-incremental to pick only files affected by my recent changes, skipping all other tests.

Currently to make use of pytest-incremental I need to first run the full test suite on my machine, which unfortunately is not viable as our tests take a lot of time, even when using xdist.

[blueyed]

@nicoddemus
Have you tried pytest-testmon in this regard?
I could imagine that syncing its .testmondata would help here.

[RonnyPfannschmidt]

i think this idea is integratable with CI artifacts
it seems sensible to upload them for releases and track them on a ci server

[nicoddemus]

@blueyed

Have you tried pytest-testmon in this regard?

I didn't, I will try it some other time, thanks for the tip.

@RonnyPfannschmidt

i think this idea is integratable with CI artifacts
it seems sensible to upload them for releases and track them on a ci server

Certainly that would be the next step.

But just to be clear, I'm asking first for a simpler solution if possible. Creating an entire mechanism of fetching and reusing CI artifacts would be a much bigger undertaking.

I was thinking of something along the lines of a --inc-consider-all-updated option which would cache all tests as "up to date" and allow the workflow I suggested (considering that this is simpler to implement than a full CI integrated solution of course).

[schettino72]

@nicoddemus doit (the underlying library used by pytest-incremental) supports this kind of workflow. In doit you need to call the command reset-dep http://pydoit.org/cmd_other.html#reset-dep before calling run.

So I guess it should not be that hard to implement this. Just need to do an extra call to doit reset-dep. Can you give it a try?

[schettino72]

@RonnyPfannschmidt I changed the title back to original because what you are asking is bit different...
It would not be hard to sync some file from a know location, actually I would like to make pytest-incremental be able to execute some generic doit tasks, so any user could easily implement something like this. Please create another issue to track this use-case.

[schettino72]

@blueyed

I know you are trying to help but you are not welcome to come to this bug tracker just to suggest using testmon... testmon doesnt even support that and you didnt really add anything to the conversation.

Also, although this project is older than testmon, it does mention it on its FAQ while testmon makes no reference to this project.

[nicoddemus]

@schettino72

So I guess it should not be that hard to implement this. Just need to do an extra call to doit reset-dep. Can you give it a try?

Sure, thanks for the tip!

Calling that command from the same directory where I usually call py.test I get this error:

$ doit reset-dep
ERROR: Could not find dodo file 'e:\ws\Souring\Projects\ben10\source\python\dodo.py'.
Please use '-f' to specify file name.

I tried then calling py.test --inc to run all tests first, but calling doit reset-dep a second time still gives me the same error.

The files created by pytest-incremental are in their usual location: deps.json and the .pytest-incremental directory.

Here's the relevant part of my environment, if it helps:

pytest==2.9.0
pytest-incremental==0.4.2
doit==0.28.0

Any other suggestion?

[schettino72]

Calling that command from the same directory where I usually call py.test I get this error:

oh. not that simple... pytest-incremental call doit in some special ways (that even I dont remember), and you will have to call reset-dep only on the "test execution" task. there are other tasks build the dependency of modules based on imports that will still need to be executed.

I would need to look at the code, and I am not 100% sure it will work...

[nicoddemus]

OK thanks for the feedback! I will try to take a look at the code as well when I have some free time.

[nicoddemus]

Investigated a little further:

$ doit reset-dep -f e:\envs\legacy-env\Lib\site-packages\pytest_incremental.py outdated
ERROR: 'outdated' is not a task.

I assumed pytest_incremental contains the tasks which usually go into a dodo.py file, due to the create_doit_tasks method:

    def create_doit_tasks(self):
        '''create all tasks used by the incremental plugin
        This method is a hook used by doit
        '''
        yield self.gen_deps()
        yield self.gen_print_deps()
        yield self.gen_dep_graph_dot()
        yield self.gen_dep_graph_image()
        yield self.gen_outdated()

(Just registering my latest attempt, will continue investigating when got some more time to spare again.)