Issue: Off-server file storage

Issue Tools
- View Changes

October 6, 2021 7:59 AM

beernuts

Junior Member

Off-server file storage

We have XF configured to store internal_data and data directories on S3 storage. A pretty common setup for attachment-heavy sites like ours. It appears that VW isn't compatible with this?

When adding a new image in the Wiki, for example at https://www.badgerandblade.com/forum...&ecid=0&eaid=2, after clicking the upload, we just get a red X, and the error log indicates:

League\Flysystem\FileNotFoundException: File not found at path: vw_attachments/2/1/3751/full.vw src/vendor/league/flysystem/src/Filesystem.php:389
#0 src/vendor/league/flysystem/src/Filesystem.php(235): League\Flysystem\Filesystem->assertPresent('vw_attachments/...')
#1 [internal function]: League\Flysystem\Filesystem->delete('vw_attachments/...', Array)
#2 src/vendor/league/flysystem-eventable-filesystem/src/EventableFilesystem.php(431): call_user_func_array('parent::delete', Array)
#3 src/vendor/league/flysystem-eventable-filesystem/src/EventableFilesystem.php(395): League\Flysystem\EventableFilesystem\EventableFilesystem->callFilesystemMethod('delete', Array)
#4 src/vendor/league/flysystem-eventable-filesystem/src/EventableFilesystem.php(330): League\Flysystem\EventableFilesystem\EventableFilesystem->delegateMethodCall('delete', Array)
#5 src/vendor/league/flysystem/src/MountManager.php(533): League\Flysystem\EventableFilesystem\EventableFilesystem->delete('vw_attachments/...')
#6 src/addons/vw/vw/_core/instance/filesystem/file/xf2.php(131): League\Flysystem\MountManager->delete('vw_attachments/...')
#7 src/addons/vw/vw/_core/instance/storage/filesystem/vw.php(180): vw_FileSystem_File_Instance_XF2->delete()
#8 src/addons/vw/vw/_core/controller/dm/attach/file/vw.php(421): vw_Storage_FileSystem_Instance->rollback(true)
#9 src/addons/vw/vw/XF/Entity/EntityBase.php(660): vw_DM_Attach_File_Controller->post_save_each()
#10 src/XF/Mvc/Entity/Entity.php(1268): vw\vw\XF\Entity\EntityBase->_postSave()
#11 src/addons/vw/vw/XF/Entity/EntityBase.php(583): XF\Mvc\Entity\Entity->save(false, false)
#12 src/addons/vw/vw/_core/controller/dm/xf2.php(390): vw\vw\XF\Entity\EntityBase->vwSave()
#13 src/addons/vw/vw/_core/controller/ui/ajax/upload/vw.php(125): vw_DM_Controller_XF2->save()
#14 src/addons/vw/vw/Pub/Controller/Wiki/Ajax.php(51): vw_UI_AJAX_Upload_Controller->execute()
#15 src/addons/vw/vw/Mvc/Dispatcher.php(30): vw\vw\Pub\Controller\Wiki\Ajax->_actionIndex(Object(XF\Mvc\ParameterBag))
#16 src/addons/vw/vw/Pub/Controller/Wiki/AbstractController.php(26): vw\vw\Mvc\Dispatcher::dispatch(Object(vw\vw\Pub\Controlle r\Wiki\Ajax), 'index', Object(XF\Mvc\ParameterBag))
#17 src/XF/Mvc/Dispatcher.php(352): vw\vw\Pub\Controller\Wiki\AbstractController->actionIndex(Object(XF\Mvc\ParameterBag))
#18 src/XF/Mvc/Dispatcher.php(259): XF\Mvc\Dispatcher->dispatchClass('vw\\vw:Wiki\\Ajax', 'Index', Object(XF\Mvc\RouteMatch), Object(vw\vw\Pub\Controller\Wiki\Ajax), NULL)
#19 src/XF/Mvc/Dispatcher.php(115): XF\Mvc\Dispatcher->dispatchFromMatch(Object(XF\Mvc\RouteMatch), Object(vw\vw\Pub\Controller\Wiki\Ajax), NULL)
#20 src/XF/Mvc/Dispatcher.php(57): XF\Mvc\Dispatcher->dispatchLoop(Object(XF\Mvc\RouteMatch))
#21 src/XF/App.php(2345): XF\Mvc\Dispatcher->run()
#22 src/XF.php(512): XF\App->run()
#23 index.php(20): XF::runApp('XF\\Pub\\App')
#24 {main}

Issue Details

Issue Number 6259

Issue Type Bug

Project VaultWiki 4.x Series

Category Unknown

Status Fixed

Priority 3 - Loss of Functionality

Affected Version 4.1.2

Fixed Version 4.1.3

Milestone (none)

Software DependencyXenForo 2.x

License TypePaid

Users able to reproduce bug 0

Users unable to reproduce bug 0

Attachments 0

Assigned Users (none)

Tags (none)

October 6, 2021 2:45 PM

pegasus

VaultWiki Team

If you created your S3 bucket before installing VaultWiki, make sure that you have uploaded the internal_data folder contents that came with VaultWiki to the internal_data folder in your S3 bucket.

Assuming you have already, I have reproduced this problem here. I believe the main issue is that we cannot establish a file lock. In src/addons/vw/vw/_core/instance/filesystem/file/xf2.php, find:

Code:

$this->lock = vw_Hard_Core::model('Lock')->lock($lockfile, 1, 'file');

Replace with:

Code:

$this->lock = vw_Hard_Core::model('Lock')->lock(1, $lockfile, 'file');

So long as you are uploading images, and those images' dimensions or file sizes exceed the wiki attachment rules, this should be enough to resolve the issue, because the new image is uploaded to AWS from memory.

But for non-images and images that are already within the rules, there are a number of other problems, such as:
- When renaming or copying files, and the source and target files are on different file systems from each other, the operation fails. This is fixed in the next release.

This takes quite a bit more work to resolve. In src/addons/vw/vw/_core/instance/filesystem/directory/xf2.php, find:

Code:

$adapter = $mount->getAdapter();

AFTER it, add:

Code:

$this->adapter = $adapter;

Find:

Code:

public function is_local_file()

BEFORE it, add:

Code:

	protected $adapter;

	public function adapter()
	{
		return $this->adapter;
	}

In src/addons/vw/vw/_core/instance/filesystem/file/xf2.php, find:

Code:

public function copy($to)

BEFORE it, add:

Code:

	protected function same_adapter($b)
	{
		$parts = explode('://', $b, 2);
		$bdapter = false;

		switch ($parts[0])
		{
			case 'data':
			case 'internal-data':
			case 'code-cache':
				$mount = XF::app()->fs()->getFilesystem($parts[0]);
				$bdapter = $mount->getAdapter();
				break;
		}

		$adapter = $this->directory->adapter();

		if ($adapter === $bdapter)
		{
			return true;
		}

		return false;
	}

Find:

Code:

		if ($this->is_local_file())
		{
			return parent::copy($to);
		}

BEFORE it, add:

Code:

		if ($this->same_adapter($to))
		{
			$stream = $this->stream();

			if (!$stream['stream'])
			{
				return false;
			}

			if (XF::app()->fs()->writeStream($to, $stream['stream']))
			{
				fclose($stream['stream']);
				$visible = XF::app()->fs()->getVisibility($this->full);
				XF::app()->fs()->setVisibility($to, $visible);

				return true;
			}
			fclose($stream['stream']);

			return false;
		}

Find:

Code:

		if ($this->is_local_file())
		{
			return parent::rename($to);
		}

BEFORE it, add:

Code:

		if ($this->same_adapter($to))
		{
			$stream = $this->stream();

			if (!$stream['stream'])
			{
				return false;
			}

			if (XF::app()->fs()->writeStream($to, $stream['stream']))
			{
				fclose($stream['stream']);
				$visible = XF::app()->fs()->getVisibility($this->full);
				XF::app()->fs()->setVisibility($to, $visible);

				$this->delete();

				return true;
			}
			fclose($stream['stream']);

			return false;
		}

After these changes, I was able to read and write VaultWiki attachments on S3.

October 7, 2021 7:18 AM

beernuts

Junior Member

Thanks for this!

I have implemented these changes, and small image uploads now work great (In the few KB range) but a larger image (Talking like 1MB here, nothing extreme) still fail. In the log, I have:

ErrorException: [E_WARNING] imagecolorallocate() expects parameter 1 to be resource, bool given src/addons/vw/vw/_core/model/image/format/vw.php:1391

#0 [internal function]: XF::handlePhpError(2, '[E_WARNING] ima...', '/home/badgerand...', 1391, Array)
#1 src/addons/vw/vw/_core/model/image/format/vw.php(1391): imagecolorallocate(true, 0, 0, 0)
#2 src/addons/vw/vw/_core/model/image/format/vw.php(1257): vw_Image_Format_Model->reorient_image(true, 6)
#3 src/addons/vw/vw/_core/controller/dm/attach/file/vw.php(940): vw_Image_Format_Model->reorient(6)
#4 src/addons/vw/vw/_core/controller/dm/attach/file/vw.php(237): vw_DM_Attach_File_Controller->set_dimensions()
#5 src/addons/vw/vw/XF/Entity/EntityBase.php(505): vw_DM_Attach_File_Controller->pre_save(true)
#6 src/XF/Mvc/Entity/Entity.php(1357): vw\vw\XF\Entity\EntityBase->_preSave()
#7 src/addons/vw/vw/XF/Entity/EntityBase.php(496): XF\Mvc\Entity\Entity->preSave()
#8 src/addons/vw/vw/_core/controller/dm/xf2.php(356): vw\vw\XF\Entity\EntityBase->vwPreSave()
#9 src/addons/vw/vw/_core/controller/ui/ajax/upload/vw.php(123): vw_DM_Controller_XF2->pre_save()
#10 src/addons/vw/vw/Pub/Controller/Wiki/Ajax.php(51): vw_UI_AJAX_Upload_Controller->execute()
#11 src/addons/vw/vw/Mvc/Dispatcher.php(30): vw\vw\Pub\Controller\Wiki\Ajax->_actionIndex(Object(XF\Mvc\ParameterBag))
#12 src/addons/vw/vw/Pub/Controller/Wiki/AbstractController.php(26): vw\vw\Mvc\Dispatcher::dispatch(Object(vw\vw\Pub\Controlle r\Wiki\Ajax), 'index', Object(XF\Mvc\ParameterBag))
#13 src/XF/Mvc/Dispatcher.php(352): vw\vw\Pub\Controller\Wiki\AbstractController->actionIndex(Object(XF\Mvc\ParameterBag))
#14 src/XF/Mvc/Dispatcher.php(259): XF\Mvc\Dispatcher->dispatchClass('vw\\vw:Wiki\\Ajax', 'Index', Object(XF\Mvc\RouteMatch), Object(vw\vw\Pub\Controller\Wiki\Ajax), NULL)
#15 src/XF/Mvc/Dispatcher.php(115): XF\Mvc\Dispatcher->dispatchFromMatch(Object(XF\Mvc\RouteMatch), Object(vw\vw\Pub\Controller\Wiki\Ajax), NULL)
#16 src/XF/Mvc/Dispatcher.php(57): XF\Mvc\Dispatcher->dispatchLoop(Object(XF\Mvc\RouteMatch))
#17 src/XF/App.php(2345): XF\Mvc\Dispatcher->run()
#18 src/XF.php(512): XF\App->run()
#19 index.php(20): XF::runApp('XF\\Pub\\App')
#20 {main}

Reply

October 7, 2021 9:57 AM

pegasus

VaultWiki Team

It looks like this occurs if you upload a file that:
- Is small enough to fit within the wiki attachment rules for file size and dimensions
- But is a landscape/portrait image that is actually stored portrait/landscape and needs to be rotated

In this case VaultWiki hasn't loaded the image data into memory but tries to rotate it.

In src/addons/vw/vw/_core/model/image/format/vw.php, find:

Code:

public function resize($x, $y, $extension, $reorient = 0)

BEFORE it, add:

Code:

	protected function assert_gd_resource()
	{
		if ($this->resource === true)
		{
			if ($this->blob)
			{
				$this->resource = @imagecreatefromstring($this->blob);
			}
			else
			{
				$this->resource = $this->create_gd_resource();
			}
		}

		if (!$this->resource)
		{
			return false;
		}

		if (isset($this->meta))
		{
			return true;
		}

		$mh = @fopen($this->get_file(), 'rb');

		if (!$mh)
		{
			return false;
		}
		fclose($mh);

		$this->init_metadata();

		return true;
	}

Find:

Code:

	public function reorient($orientation, $metadata = null)
	{

AFTER it, add:

Code:

		if ($this->gd AND !$this->assert_gd_resource())
		{
			return '';
		}

October 7, 2021 10:01 AM

beernuts

Junior Member

Thank you for the amazing support, as always!

I can confirm that this has fixed the issue.

Reply
October 11, 2021 6:25 PM

beernuts

Junior Member

Possibly related to this new code, I got this from one of the serious wiki users:

I did a few attachment uploads and noticed one thing. There seems to be a small glitch where I need to edit the page and save it again for the newly uploaded image to work.

It works but if you do upload something, you may need to edit and save it again. Since it works, I wouldn't worry much for now.

Reply
October 12, 2021 4:52 AM

pegasus

VaultWiki Team

Unfortunately that report is a bit vague (discussion of making something "work" without describing the expected behavior, "working," vs the actual behavior, "not working"). My guess is that it's related to another issue we are aware of for attachments, which sometimes causes attachment pages to become inaccessible ("node does not exist" message). That issue is slated to be fixed in the next round of patches.

Reply
October 12, 2021 7:20 AM

beernuts

Junior Member

Yeah, I know it's vague, sorry. I don't know enough about the wiki to know what to ask for in clarification. We'll just leave this alone and see if it resolves itself with the next update.
Thanks

Reply
November 1, 2021 9:35 AM

beernuts

Junior Member

Seems like there is still an issue, this time relating to using ImageMagick instead of GD. We just made the switch last week and now get this in the logs when someone tries to upload an image to the Wiki:

#0 src/addons/vw/vw/_core/model/image/processor/imagick/vw.php(36): vw_Image_Processor_Imagick_Model::get_temppath_prefix()
#1 src/addons/vw/vw/_core/model/image/processor/imagick/vw.php(28): vw_Image_Processor_Imagick_Model::get_sandbox_path()
#2 src/addons/vw/vw/_core/model/image/format/vw.php(1654): vw_Image_Processor_Imagick_Model->__construct('/tmp/phpH6cj4i')
#3 src/addons/vw/vw/_core/model/image/format/vw.php(1632): vw_Image_Format_Model->init_resource()
#4 src/addons/vw/vw/_core/model/image/format/vw.php(578): vw_Image_Format_Model->set_filedata()
#5 src/addons/vw/vw/_core/controller/dm/attach/file/vw.php(783): vw_Image_Format_Model->get_width()
#6 src/addons/vw/vw/_core/controller/dm/attach/file/vw.php(237): vw_DM_Attach_File_Controller->set_dimensions()
#7 src/addons/vw/vw/XF/Entity/EntityBase.php(505): vw_DM_Attach_File_Controller->pre_save(true)
#8 src/XF/Mvc/Entity/Entity.php(1357): vw\vw\XF\Entity\EntityBase->_preSave()
#9 src/addons/vw/vw/XF/Entity/EntityBase.php(496): XF\Mvc\Entity\Entity->preSave()
#10 src/addons/vw/vw/_core/controller/dm/xf2.php(356): vw\vw\XF\Entity\EntityBase->vwPreSave()
#11 src/addons/vw/vw/_core/controller/ui/ajax/upload/vw.php(123): vw_DM_Controller_XF2->pre_save()
#12 src/addons/vw/vw/Pub/Controller/Wiki/Ajax.php(51): vw_UI_AJAX_Upload_Controller->execute()
#13 src/addons/vw/vw/Mvc/Dispatcher.php(30): vw\vw\Pub\Controller\Wiki\Ajax->_actionIndex(Object(XF\Mvc\ParameterBag))
#14 src/addons/vw/vw/Pub/Controller/Wiki/AbstractController.php(26): vw\vw\Mvc\Dispatcher::dispatch(Object(vw\vw\Pub\Controlle r\Wiki\Ajax), 'index', Object(XF\Mvc\ParameterBag))
#15 src/XF/Mvc/Dispatcher.php(352): vw\vw\Pub\Controller\Wiki\AbstractController->actionIndex(Object(XF\Mvc\ParameterBag))
#16 src/XF/Mvc/Dispatcher.php(259): XF\Mvc\Dispatcher->dispatchClass('vw\\vw:Wiki\\Ajax', 'Index', Object(XF\Mvc\RouteMatch), Object(vw\vw\Pub\Controller\Wiki\Ajax), NULL)
#17 src/XF/Mvc/Dispatcher.php(115): XF\Mvc\Dispatcher->dispatchFromMatch(Object(XF\Mvc\RouteMatch), Object(vw\vw\Pub\Controller\Wiki\Ajax), NULL)
#18 src/XF/Mvc/Dispatcher.php(57): XF\Mvc\Dispatcher->dispatchLoop(Object(XF\Mvc\RouteMatch))
#19 src/XF/App.php(2345): XF\Mvc\Dispatcher->run()
#20 src/XF.php(512): XF\App->run()
#21 index.php(20): XF::runApp('XF\\Pub\\App')
#22 {main}

Error: Call to a member function is_local() on string src/addons/vw/vw/_core/model/image/processor/imagick/vw.php:120

Reply
November 1, 2021 10:49 AM
pegasus

VaultWiki Team
In src/addons/vw/vw/_core/model/image/processor/imagick/vw.php, find all:

Code:

if ($path->is_local())

Replace all with:

Code:

if ($path AND $path->is_local())
Reply
November 3, 2021 8:32 AM

beernuts

Junior Member

Thank you. I believe that fixed that. I have one more report from the wiki guys.

I believe there's one more bug.

When we upload a picture to the image space, it saves without any error. The picture doesn't work when we code it in a wiki page. The uploaded picture needs to be edited and saved again to work.

Any thoughts on this one?

Reply
November 3, 2021 11:30 AM
pegasus

VaultWiki Team
It's hard to say, since "the picture doesn't work" doesn't describe the behavior that the user is actually seeing (FILE renders as expected but console shows error when trying to load the image? FILE displays a 0x0 image? FILE displays error message? FILE displays a red link? FILE renders working link but not image? FILE renders as empty string? anything like this meanwhile image space location renders complete image?)

I see a few items in the 4.1.3 changelog that could be related, such as:

Code:

Fixed Bug: thumbnail generated from original file location after file moved [lite: no] Fixed Bug: thumbnails instantly deleted after creation [since: 4.1.0 Alpha 1] [lite: no]

There were also some items in the October patches which may be related. I would encourage you to upgrade to the latest version and see if that resolves the issue. 4.1.3 already includes all the fixes that were mentioned earlier in this thread.

If upgrading does not resolve the issue, please take the following steps:
- The next time that a picture is uploaded and it exhibits the problematic behavior, do not fix it. Instead, report the specific picture in the image space to me. Also report the wiki page that uses the picture to me. Again, do not fix it. I need to see what the actual behavior is, and I need to examine the metadata. Examining the metadata will require that you submit a ticket, because I would need to access the files that process that metadata.
Reply

+ Reply

All times are GMT -4. The time now is 2:43 AM.

This site uses cookies to help personalize content, to tailor your experience, and to keep you logged in if you register.
By continuing to use this site, you are consenting to our use of cookies.

Learn more… Accept Remind me later

Welcome to VaultWiki.org, home of the wiki add-on for vBulletin and XenForo!

Issue: Off-server file storage

Issue Tools