Discussion:
xdg-utils does not implement the standard for default applications
Maxwell Anselm
2016-10-03 20:09:50 UTC
Permalink
Regarding the standard for the association between MIME types and
applications (https://specifications.freedesktop.org/mime-apps-spec/latest/),
xdg-utils provides the tools xdg-mime (for querying a file's MIME type) and
xdg-open (for opening applications based on MIME-type).

xdg-mime and xdg-open both attempt to detect the user's desktop environment
and then delegate to some DE-specific tool e.g. gvs-info, kde-open, etc.
That is fine and works well for DE integration. However, in the absence of
a DE, these tools fall back to methods which result in nonstandard behavior.

For example, xdg-mime tries to use mimetype (implemented with Perl's
File::MimeInfo) and if that does not exist, falls back to plain old
/usr/bin/file. Similarly xdg-open tries to use mimeopen (also from Perl)
before anything else. Why is this a problem?

1. While File::MimeInfo may intend to implement the standard, its
developers have no obligation (and seemingly no interest) in doing so well.
mimetype and mimeopen each have several trivially fixable bugs that have
gone unpatched for years: https://github.com/mbeijen/File-MimeInfo/issues
2. /usr/bin/file in particular makes no attempt to follow the XDG standard.
It makes absolutely no sense as a fallback because it will misclassify all
but the most mundane MIME types
3. A lack of a "reference implementation" has led to extreme fragmentation.
Arch Linux alone has six unofficial xdg-open replacements developed by
users who were tired of figuring out why their default application choices
were not being respected by xdg-open.

Let me give you a real example that I experienced (and which sparked my
investigation of this matter): if you do not run a DE, it is entirely
likely that xdg-mime will store its MIME/application associations in a
deprecated location and that xdg-open will completely ignore these
associations (deprecated or otherwise) and open a completely different
application. That simply should not be possible with tools provided by the
organization that has developed the standard.

The freedesktop community would benefit considerably from XDG providing an
actual implementation of the default application standard within xdg-mime
and xdg-open. This would benefit the small but significant group of users
who do not run a well-known DE, this would provide a standard to which
alternative implementations could be compared, and it would place the
ultimate responsibility of standard compliance in the hand of the
standardizing organization (rather than some random CPAN developer). To put
it concretely:

1. xdg-mime and xdg-open should not rely on Perl's File::MimeInfo at all
(this behavior could be enabled with an optional flag if you don't want to
just drop it).
2. In the absence of a detected DE, xdg-mime should be fully capable of
following the shared MIME-info spec to detect a file's MIME type:
https://specifications.freedesktop.org/shared-mime-info-spec/latest/ar01s02.html
3. In the absence of a detected DE, xdg-mime should be fully capable of
creating associations and storing them in the standard, non-deprecated
locations:
https://specifications.freedesktop.org/mime-apps-spec/latest/ar01s02.html
4. In the absence of a detected DE, xdg-open should simply use xdg-mime to
query the MIME type and then use the same standard locations (plus
deprecated, for compatibility) to look up the default application for that
type

I am willing to help develop these changes, but only if it is clear that
such contributions would actually be welcome and accepted.

Max
Reuben Thomas
2016-10-03 20:20:22 UTC
Permalink
Post by Maxwell Anselm
I am willing to help develop these changes, but only if it is clear that
such contributions would actually be welcome and accepted.
​This would be great to see.

​As an occasional xdg-utils contributor and sometime /usr/bin/file
maintainer, I'd say that it would be good to see more work on xdg-utils;
and that I think I can speak for file maintainers when I say that if file
can, within its obvious limitations (specifically, that it only identifies
files by their contents), ​be improved, patches would be speedily processed.

I'm unclear whether you think that /usr/bin/file *should* not be used by
xdg-mime (in which case, fair enough, I guess), or simply that it currently
provides the wrong types or mis-identifies files too readily.

Wrong MIME types in file's patterns can of course easily be fixed (I added
them only relatively recently).

Mis-identification is harder to fix, of course, but improved patterns are
always welcome, and in the absence of other methods of identification, it
seems to me a better fallback than nothing. It could perhaps be improved if
there were some way for xdg-mime to indicate a confidence level, as file
itself has "strength" levels internally which could be used to indicate a
degree of confidence, and, with a little work, it could for example not
return a match if it was below a certain level of confidence.​

--
​ ​

http://rrt.sc3d.org
Maxwell Anselm
2016-10-04 19:34:33 UTC
Permalink
Post by Reuben Thomas
I'm unclear whether you think that /usr/bin/file *should* not be used by
xdg-mime (in which case, fair enough, I guess), or simply that it currently
provides the wrong types or mis-identifies files too readily.
file is a fine tool for what it intends to do, but since it does *not*
intend to implement the standard it does not make sense as a fallback. For
example, the shared MIME-info database allows for specifying MIME type
according to the internal root and namespace of an XML file. Unless the
file devs are willing to push an update for every single esoteric XML
format out there, xdg-mime could end up behaving incorrectly.

If xdg-mime and xdg-open implement the standard themselves there would be
no need for any external fallback.

Max
Reuben Thomas
2016-10-04 19:39:31 UTC
Permalink
Post by Maxwell Anselm
Post by Reuben Thomas
I'm unclear whether you think that /usr/bin/file *should* not be used by
xdg-mime (in which case, fair enough, I guess), or simply that it currently
provides the wrong types or mis-identifies files too readily.
file is a fine tool for what it intends to do, but since it does *not*
intend to implement the standard it does not make sense as a fallback. For
example, the shared MIME-info database allows for specifying MIME type
according to the internal root and namespace of an XML file. Unless the
file devs are willing to push an update for every single esoteric XML
format out there, xdg-mime could end up behaving incorrectly.
If xdg-mime and xdg-open implement the standard themselves there would be
no need for any external fallback.
Unless I've misunderstood something, there might well be file formats
formats that file can identify which another implementation cannot, even if
xdg-mime and xdg-open do implement the standard, simply because of the
range of formats covered by their database. So the use of file could still
be complementary.

There's also some scope for getting file to pull namespaces out of XML
files. I'm not sure what you mean by an "esoteric XML format"; surely
there's only one definition of XML? Or are you saying that such XML files
would not actually contain their MIME type, so it would still have to be
recognised somehow?
--
http://rrt.sc3d.org
Maxwell Anselm
2016-10-04 20:05:02 UTC
Permalink
Post by Reuben Thomas
Unless I've misunderstood something, there might well be file formats
formats that file can identify which another implementation cannot, even if
xdg-mime and xdg-open do implement the standard, simply because of the
range of formats covered by their database. So the use of file could still
be complementary.
There's also some scope for getting file to pull namespaces out of XML
files. I'm not sure what you mean by an "esoteric XML format"; surely
there's only one definition of XML? Or are you saying that such XML files
would not actually contain their MIME type, so it would still have to be
recognised somehow?
I see what you mean about file as a fallback in that sense. That is a
reasonable use if the MIME-info database has no information relating to the
file. Currently, file is a fallback in the sense that if xdg-mime does not
detect a DE and mimetype is not installed, then it will use file to get the
MIME type without even attempting to check the MIME-info database itself.
That is what should be changed.

For a specific example of what I'm talking about with XML, I package an RPG
utility for Arch Linux called gcs. gcs mainly uses XML files for its data,
for example *.spl files are XML files with a spell_list root element that
store magic spell information. If I want it to be possible to double click
one of these files and have gcs open as the default application, I need to
register a new MIME type: gcs obviously can't handle arbitrary XML files,
and *.spl is also used for Shockwave Flash files. Now if the file devs
really want to push an update so that it correctly identifies gcs spell
list files, go right ahead, but personally I would find that kind of
ridiculous for such an obscure and specialized utility (though if you do,
please use application/x-gurps-spell so that I don't have to update my
package ;P).
Reuben Thomas
2016-10-04 21:07:39 UTC
Permalink
Post by Maxwell Anselm
For a specific example of what I'm talking about with XML, I package an
RPG utility for Arch Linux called gcs. gcs mainly uses XML files for its
data, for example *.spl files are XML files with a spell_list root element
that store magic spell information. If I want it to be possible to double
click one of these files and have gcs open as the default application, I
need to register a new MIME type: gcs obviously can't handle arbitrary XML
files, and *.spl is also used for Shockwave Flash files. Now if the file
devs really want to push an update so that it correctly identifies gcs
spell list files, go right ahead, but personally I would find that kind of
ridiculous for such an obscure and specialized utility (though if you do,
please use application/x-gurps-spell so that I don't have to update my
package ;P).
​Generally new patterns are added as they're supplied. File can detect all
sorts of funny formats, and there's no criterion of importance. Please do
supply some patterns, and, especially for an unusual format, test files!

Ways to contribute listed here: http://www.darwinsys.com/file/ (bug tracker
or mailing list)​
--
http://rrt.sc3d.org
Maxwell Anselm
2016-10-04 23:08:25 UTC
Permalink
I appreciate the sentiment, but frankly this isn't about improving file.
Even if file could identify every file in existence correctly, a user could
still provide a MIME type override on their system and get nonstandard,
surprising behavior from xdg-mime when it ignores the MIME info databases
and just calls file.

Can an actual xdg-utils maintainer way in on this?
Post by Reuben Thomas
Post by Maxwell Anselm
For a specific example of what I'm talking about with XML, I package an
RPG utility for Arch Linux called gcs. gcs mainly uses XML files for its
data, for example *.spl files are XML files with a spell_list root element
that store magic spell information. If I want it to be possible to double
click one of these files and have gcs open as the default application, I
need to register a new MIME type: gcs obviously can't handle arbitrary XML
files, and *.spl is also used for Shockwave Flash files. Now if the file
devs really want to push an update so that it correctly identifies gcs
spell list files, go right ahead, but personally I would find that kind of
ridiculous for such an obscure and specialized utility (though if you do,
please use application/x-gurps-spell so that I don't have to update my
package ;P).
​Generally new patterns are added as they're supplied. File can detect all
sorts of funny formats, and there's no criterion of importance. Please do
supply some patterns, and, especially for an unusual format, test files!
Ways to contribute listed here: http://www.darwinsys.com/file/ (bug
tracker or mailing list)​
--
http://rrt.sc3d.org
David Faure
2016-10-29 16:25:32 UTC
Permalink
Post by Maxwell Anselm
1. xdg-mime and xdg-open should not rely on Perl's File::MimeInfo at all
(this behavior could be enabled with an optional flag if you don't want to
just drop it).
2. In the absence of a detected DE, xdg-mime should be fully capable of
https://specifications.freedesktop.org/shared-mime-info-spec/latest/ar01s02.
html 3. In the absence of a detected DE, xdg-mime should be fully capable of
creating associations and storing them in the standard, non-deprecated
https://specifications.freedesktop.org/mime-apps-spec/latest/ar01s02.html
4. In the absence of a detected DE, xdg-open should simply use xdg-mime to
query the MIME type and then use the same standard locations (plus
deprecated, for compatibility) to look up the default application for that
type
As a co-maintainer of the shared-mime-info spec and other specs (but not of
any of the xdg-* tools), I fully agree with your reasoning.

`xdg-mime query` could implement shared-mime-info rather than trying to find an
existing implementation. Your point 1 would be even extended further: xdg-mime
would not rely on my kmimetypefinder or on gnomevfs-info/gvfs-info either,
since it would simply call a tool that implements the spec.

Code exists to do this: see print-mime-data in the git repo "git/xdg/xdgmime".
(of course you need to tweak its input arguments and output format so that it
can be used by `xdg-mime query`).

However that repo is separate from xdg-utils, so you could end up with the
issue that xdg-utils is installed but not xdgmime. I think the best approach
would be to merge these two git repos.
(CC'ing the most recent committers to xdg-utils in case they have input)
--
David Faure, ***@kde.org, http://www.davidfaure.fr
Working on KDE Frameworks 5
Loading...