docgen as a preprocessor, fixes #193

[seanhess]

Switched docgen to be a preprocessor, and updated build-tools to depend on it. This works out of the box for cabal build and cabal haddock, but unfortunately requires manaully installing the preprocessor to your PATH for HLS to work. I can't think of a workaround.

I suppose we could easily add a line to the developer instructions in the README to remind people to run:

cabal install docs:docgen

[seanhess]

@sectore what needs to happen to get a local preprocessor working for nix? Note that cabal build works without any extra steps, but HLS fails unless you manually install it:

cabal install docgen:docgen

[sectore]

Just FYI: With latest commits in current PR, Nix users will have docgen on PATH by entering nix develop shell by default. Not sure if its worth to mention it in dev.md as this document is for non Nix users only.

[sectore]

Look at this ⬇️ Nix builds take only 1-3min now. Cachix does a nice job.

[sectore]

Please double check packcheck/ci-sdist ..., which is still failing (here). It expects docgen:docgen, but that's a package available locally only?

[seanhess]

I don't understand why sdist is failing. Packcheck is supposed to be reproducable when you run it locally, but it works absolutely fine on my laptop. I can't figure out why the sdist check would handle docgen differently

Just FYI: With latest commits in current PR, Nix users will have docgen on PATH by entering nix develop shell by default. Not sure if its worth to mention it in dev.md as this document is for non Nix users only.

That's great! I'll add a note that it's only for non-nix users. Actually, it probably makes sense to have completely separate nix and non-nix dev documents. I assume other things in dev.md don't apply? Certainly not brew install npm.

[seanhess]

Grrr, it doesn't look like this is going to work: composewell/packcheck#121 (comment)

[sectore]

Grrr, it doesn't look like this is going to work: composewell/packcheck#121 (comment)

Nice!

Actually, it probably makes sense to have completely separate nix and non-nix dev documents. I assume other things in dev.md don't apply? Certainly not brew install npm.

Yes and yes.

Please try to bring commits 13af33c and 4242287 back into this branch. It seems they are lost by a previous force push. They are parts of my previous commits I have done before here.

13af33c (HEAD -> docgen, origin/docgen) nix: `docgen` on PATH
4242287 fix(nix): preprocessor failed

[seanhess]

Please try to bring commits 13af33c and 4242287 back into this branch. It seems they are lost by a previous force push. They are parts of my previous commits I have done before here.

Oh, I'm sorry, I didn't realize you had pushed to this branch. I was confused how I got out of sync.

Nice!

Ha, no, it's bad news. I'm not sure the local pre-processor approach will work at all. Maybe the right thing to do is to publish the preprocessor to hackage or research how other people are accomplishing the same thing. Having type-checked examples in your haddock is incredibility useful, I can't be the only one who wants to do that.

[seanhess]

Our options are as follows: (See this reddit thread)

1. Switch to using doctest
2. Publish docgen to hackage

Doctest is more standard, and we don't have to publish anything. Our docgen produces better output, and doesn't break HLS editor integration to find errors in the examples.

I'm leaning towards publishing docgen to hackage, leaving it specific to hyperbole, and namespacing it as hyperbole-docgen. That's the least amount of work, and I really like having the examples typechecked in real haskell modules

[seanhess]

@sectore Could you help me repair the nix config? I decided we should publish hyperbole:docgen. Local builds can still depend on it via the cabal.project file, while sdist can rely on the published version.

Specifically I changed:

1. Removed docs/docgen.cabal (via package.yaml)
2. Added docgen executable to hyperbole.cabal
3. Renamed docs/Main.hs to docs/Docgen.hs
4. Removed ./docs from cabal.project, since it is now local

[seanhess]

@sectore See above. I don't want this branch to get too out of date before merging

[sectore]

@sectore Could you help me repair the nix config?

Do current changes work for you? packcheck/ci-sdist seems still to be failing at CI.

I have tried to fix Nix, but it's not that easy... Now we have a docgen executable defined in hyperbole.cabal pointing to another folder. Besides that a symlink approach of docs folder to be available in demo ... Nix goes another way than Cabal.

Before I continue to try to make Nix work, could you make packcheck/ci-sdist running "green" first, pls?

[seanhess]

Do current changes work for you? packcheck/ci-sdist seems still to be failing at CI.

That's expected. With the current approach, it can't be fixed until we cut a release. It needs a hyperbole:docgen published to hackage.

[sectore]

With the current approach, it can't be fixed until we cut a release. It needs a hyperbole:docgen published to hackage.

Why not publishing a simple package hyperbole-docgen to Hackage (instead of adding it hyperbole)?

No another hyperbole release needed then. packcheck/ci-sdist can then be tested easily at CI, too. Also things on Nix side will be simplified by just loading such a docgen package from Hackage.

[seanhess]

Why not publishing a simple package hyperbole-docgen to Hackage (instead of adding it hyperbole)?

No another hyperbole release needed then. packcheck/ci-sdist can then be tested easily at CI, too. Also things on Nix side will be simplified by just loading such a docgen package from Hackage.

It feels like we would need to clean it up to make it general enough to be useful to other people as a separate package. Then we have to remember to release it, etc. Having it on the hyperbole package as hyperbole:docgen would work exactly the same way as a published hyperbole-docgen:docgen executable. It would be overkill to fix CI one time, when publishing hyperbole once would fix the problem. We're close to a release anyway.

Does nix have the same problem as sdist? I thought we had it working against the local version before. I just don't know how to update it for having it in the hyperbole package instead of a sub directory.

[sectore]

Does nix have the same problem as sdist?

With latest commits Nix builds and provides docgen executable as needed.

which docgen
/nix/store/l7wr55markxx2q8diiza41n5hbs3i7n9-ghc-9.10.3-with-packages/bin/docgen

PLEASE don't force-push anything in current branch Just to keep changes intact lol...

In case HLS in your editor can render documentations and example code properly, can you share a screen shot, pls?

It seems my editor (Zed) can't render the examples, but it might be an issue of Zed's Haskell extension (not of current Nix builds).

[seanhess]

PLEASE don't force-push anything in current branch Just to keep changes intact lol...

LOL, sorry about that. I'm not used to the workflow of someone else pushing to my PR branch, and I have a habit of rebasing branches. I'll be more careful.

In case HLS in your editor can render documentations and example code properly, can you share a screen shot, pls?

I'm not 100% sure what you are asking for. To get HLS to work in my editor I have to install hyperbole:docgen to my PATH:

cabal install hyperbole:docgen --overwrite-policy=always

If my editor is failing, it is helpful to manually run HLS:

haskell-language-server-wrapper-2.13.0.0 src/Web/Hyperbole.hs

[sectore]

can you share a screen shot, pls?

I'm not 100% sure what you are asking for.

Not really important as I was just wondering how a popup look like in your editor (IIRC Neovim?) by hovering variables and functions in your code. That popup should include it's type definition and (most interesting to me) the documentation we get with docgen now.

Ready to merge that beast (current PR)?

[seanhess]

Not really important as I was just wondering how a popup look like in your editor (IIRC Neovim?) by hovering variables and functions in your code. That popup should include it's type definition and (most interesting to me) the documentation we get with docgen now.

Oh I see. I've never configured that. I do have goto definition working, so the linking is there. Documentation hovering would be nice!

Ready to merge that beast (current PR)?

Let's do it!