Mass Import and Overwrite Pages?

[SurferJon]

Is there a way to make text files of pages/templates, then mass import them to create those pages/templates? Furthermore, would there be a way to completely overwrite said pages/templates if they already exist? This wouldn't be a one-time operation, I would be doing it all the time.

I know there's an import/export option, so I was thinking I could trick it to do what I'm describing? Is there some kind of format I could use to create my pages/templates locally, then use the import tool to put my locally created pages/templates in the wiki?

It wouldn't matter if it wipes out the edit history for the pages/templates it's overwriting.

I'm using the XF VW.

Thank you!

[pegasus]

You would have to write a custom import tool to do this. I'm not sure why you would do the work to create those text files rather than just create the pages directly, though?

Perhaps you might explain a little more what your idea is and how the workflow would work.

[SurferJon]

You would have to write a custom import tool to do this. I'm not sure why you would do the work to create those text files rather than just create the pages directly, though?

Perhaps you might explain a little more what your idea is and how the workflow would work.

So basically my wiki deals with a lot of video game and trading card data. My wiki is half database, half user-written material. Each time a new game or set comes out, hundreds of character data templates need to be updated simultaneously and within minutes so that all pages are updated. I basically extract the data on my computer, and then I would put it into the format I need for VW and upload it so everything gets updated simultaneously.

[pegasus]

If you are good with PHP, basically you want to create:
- vault/core/controller/import/steps/customtextfile/vw.php
- vault/core/controller/import/handle/customtextfile/vw.php

Your steps file would probably only define one step, which runs a method in the handle file.

Your handle file would parse the text file you uploaded and pass the data to the wiki Page DM.

Right now there is not a standardized VaultWiki XML format or something like that, although that has been one of the ideas thrown around to add to the importer with a corresponding export function.

[SurferJon]

How difficult/time consuming do you think this would be for a programmer to do? Is most of the programming already in there, they just need to customize it to accept some kind of external file with the pages I want to add? Or would we not be able to use the existing functions since they are only for new wiki imports and overwrite the whole wiki? Thank you!

[pegasus]

Imports are not designed to overwrite the whole wiki.

A programmer would need to:
- Write a steps file that breaks down your import into steps, such as import pages, import comments, etc. There is already support to do each step in batches (like if step 1 would add 1,000,000 pages, best to do this in batches) if the developer wants to implement that. If you only need one step, then the steps file only needs one step in it.
- Write handlers for your steps that parse your text file and walk through the data. The handler should pass the data to the appropriate DM in order to create the wiki content.
- There is a built-in mapping system for quickly looking up the IDs of imported data based on the source ID. This is useful when you want to make sure an imported comment is connected to the correct imported page, since the IDs in your wiki won't match the IDs of the source data. The developer just needs to implement the writes/reads for the mapper. For a text-file-based import, the developer will probably want to override the hashing function that creates a unique ID for each import; it should probably use a hash of the external file instead of source database credentials.

A good developer should be able to get a sense of how to put an importer together by looking at the /vault/core/controller/import/steps and /vault/core/controller/import/handle folders for some examples. It only gets confusing if you try to figure out how the progress tracker works, which no one should have to do. VaultWiki does this for you if your step file is written well.

[mon169693]

Thank pegasus, Your post very good.