Skip to content

Commit f69fd55

Browse files
livecodepanoslivecodepanos
authored
Merge pull request livecode#7238 from montegoulding/bugfix-22213
[[ Bug 22213 ]] Implement detailed-utf8 files & folders
2 parents ba91056 + ca52979 commit f69fd55

File tree

9 files changed

+97
-20
lines changed

9 files changed

+97
-20
lines changed

‎docs/dictionary/function/files.lcdoc‎

Lines changed: 8 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -41,7 +41,9 @@ The folder path to search.
4141

4242
outputKind (enum):
4343
- empty: Use the short form
44-
- "detailed": Use the long/detailed form
44+
- "detailed": Use the long/detailed form (deprecated)
45+
- "detailed-utf8": Use the long/detailed form encoding file names as UTF8 before
46+
URL-encoding.
4547

4648
Returns (String):
4749
A list of file names or file information, with one
@@ -84,8 +86,9 @@ value is a list of files with detailed file attributes.
8486
Each line in the return value is a comma-separated list of file
8587
attributes, as follows:
8688

87-
1. **Name**. The file name is URL-encoded. To obtain the name as
88-
plain text, use the <URLDecode> function.
89+
1. **Name**. The file name is URL-encoded. To reliably obtain the name use the
90+
`detailed-utf8` <outputKind> and decode using the <URLDecode> function followed
91+
by <textDecode>. For example, `textDecode(URLDecode(item 1 of tLine), "utf8")`.
8992
2. **Size**, in bytes. On OS X systems, this is the size of the
9093
<data fork>.
9194
3. **Resource fork size**, in bytes. (OS X only).
@@ -111,13 +114,14 @@ Changes:
111114
The <long> form was introduced in version 1.1.
112115
The optional <targetFolder> argument for the function call form was
113116
introduced in version 8.1.
117+
The `detailed-utf8` form was introduced in version 9.6.
114118

115119
References: sort (command), return (constant), folders (function),
116120
specialFolderPath (function), alias (glossary), current folder (glossary),
117121
data fork (glossary), folder (glossary), function (glossary),
118122
platform (glossary), return (glossary), resource fork (glossary),
119123
shortcut (glossary), symbolic link (glossary), defaultFolder (property),
120-
umask (property), long (keyword)
124+
umask (property), long (keyword), URLDecode (function), textDecode (function)
121125

122126
Tags: file system
123127

‎docs/dictionary/function/folders.lcdoc‎

Lines changed: 10 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -35,7 +35,9 @@ The folder path to search.
3535

3636
outputKind (enum):
3737
- empty: Use the short form
38-
- "detailed": Use the long/detailed form
38+
- "detailed": Use the long/detailed form (deprecated)
39+
- "detailed-utf8": Use the long/detailed form encoding folder names as UTF8
40+
before URL-encoding.
3941

4042
Returns (String):
4143
A list of folder names or folder information, with
@@ -91,8 +93,10 @@ the attributes are provided only for compatibility with the <files>
9193
Each line in the return value is a comma-separated list of file
9294
attributes, as follows:
9395

94-
1. **Name**. The folder name is URL-encoded. To obtain the name as
95-
plain text, use the <URLDecode> function.
96+
1. **Name**. The folder name is URL-encoded. To reliably obtain the name use
97+
the `detailed-utf8` <outputKind> and decode using the <URLDecode> function
98+
followed by <textDecode>. For example,
99+
`textDecode(URLDecode(item 1 of tLine), "utf8")`.
96100
2. **Size**, in bytes. This is always 0.
97101
3. **Resource fork size**, in bytes. This is always 0.
98102
4. **Creation date**, in seconds. (OS X and Windows only).
@@ -119,13 +123,15 @@ Changes:
119123
The <long> form was introduced in version 1.1.
120124
The optional <targetFolder> argument for the function call form was
121125
introduced in version 8.1.
126+
The `detailed-utf8` form was introduced in version 9.6.
122127

123128
References: sort (command), return (constant), folders (function),
124129
specialFolderPath (function), alias (glossary), current folder (glossary),
125130
data fork (glossary), folder (glossary), function (glossary),
126131
platform (glossary), return (glossary), resource fork (glossary),
127132
shortcut (glossary), subfolder (glossary), symbolic link (glossary),
128-
defaultFolder (property), umask (property), long (keyword)
133+
defaultFolder (property), umask (property), long (keyword),
134+
URLDecode (function), textDecode (function)
129135

130136
Tags: file system
131137

‎docs/notes/bugfix-22213.md‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
# A new output kind `detailed-utf8` has been added to the `files` and `folders` functions
2+
3+
The `{ long | detailed } { files | folders } of <folder>`,
4+
`files(<folder>, "detailed")`, and `folders(<folder>, "detailed")` suffer from
5+
an anomaly bug where file and folder names are native encoded before being
6+
URL encoded to add to the list. The native encoding is can not represent
7+
many unicode codepoints and is therefore lossy.
8+
9+
The new `detailed-utf8` output kind encodes file and folder names as utf8 before
10+
URL encoding them. This allows the names to be decoded via
11+
`textDecode(URLDecode(<name>), "utf8")`.

‎engine/src/exec-files.cpp‎

Lines changed: 6 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -58,6 +58,7 @@ MCFilesEvalFileItemsOfDirectory(MCExecContext& ctxt,
5858
MCStringRef p_directory,
5959
bool p_files,
6060
bool p_detailed,
61+
bool p_utf8,
6162
MCStringRef& r_string)
6263
{
6364
if (MCsecuremode & MC_SECUREMODE_DISK)
@@ -66,7 +67,7 @@ MCFilesEvalFileItemsOfDirectory(MCExecContext& ctxt,
6667
return;
6768
}
6869
MCAutoListRef t_list;
69-
if (MCS_getentries(p_directory, p_files, p_detailed, &t_list))
70+
if (MCS_getentries(p_directory, p_files, p_detailed, p_utf8, &t_list))
7071
{
7172
MCListCopyAsString(*t_list, r_string);
7273
}
@@ -2716,7 +2717,7 @@ void MCFilesGetFiles(MCExecContext& ctxt, MCStringRef& r_value)
27162717
if (ctxt . EnsureDiskAccessIsAllowed())
27172718
{
27182719
MCAutoListRef t_list;
2719-
if (MCS_getentries(nil, true, false, &t_list) && MCListCopyAsString(*t_list, r_value))
2720+
if (MCS_getentries(nil, true, false, false, &t_list) && MCListCopyAsString(*t_list, r_value))
27202721
return;
27212722

27222723
ctxt . Throw();
@@ -2728,7 +2729,7 @@ void MCFilesGetDetailedFiles(MCExecContext& ctxt, MCStringRef& r_value)
27282729
if (ctxt . EnsureDiskAccessIsAllowed())
27292730
{
27302731
MCAutoListRef t_list;
2731-
if (MCS_getentries(nil, true, true, &t_list) && MCListCopyAsString(*t_list, r_value))
2732+
if (MCS_getentries(nil, true, true, false, &t_list) && MCListCopyAsString(*t_list, r_value))
27322733
return;
27332734

27342735
ctxt . Throw();
@@ -2740,7 +2741,7 @@ void MCFilesGetFolders(MCExecContext& ctxt, MCStringRef& r_value)
27402741
if (ctxt . EnsureDiskAccessIsAllowed())
27412742
{
27422743
MCAutoListRef t_list;
2743-
if (MCS_getentries(nil, false, false, &t_list) && MCListCopyAsString(*t_list, r_value))
2744+
if (MCS_getentries(nil, false, false, false, &t_list) && MCListCopyAsString(*t_list, r_value))
27442745
return;
27452746

27462747
ctxt . Throw();
@@ -2752,7 +2753,7 @@ void MCFilesGetDetailedFolders(MCExecContext& ctxt, MCStringRef& r_value)
27522753
if (ctxt . EnsureDiskAccessIsAllowed())
27532754
{
27542755
MCAutoListRef t_list;
2755-
if (MCS_getentries(nil, false, true, &t_list) && MCListCopyAsString(*t_list, r_value))
2756+
if (MCS_getentries(nil, false, true, false, &t_list) && MCListCopyAsString(*t_list, r_value))
27562757
return;
27572758

27582759
ctxt . Throw();

‎engine/src/exec.h‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3158,7 +3158,7 @@ void MCEngineSetRevLibraryMappingByKey(MCExecContext& ctxt, MCNameRef p_library,
31583158

31593159
///////////
31603160

3161-
void MCFilesEvalFileItemsOfDirectory(MCExecContext& ctxt, MCStringRef p_directory, bool p_files, bool p_detailed, MCStringRef& r_string);
3161+
void MCFilesEvalFileItemsOfDirectory(MCExecContext& ctxt, MCStringRef p_directory, bool p_files, bool p_detailed, bool p_utf8, MCStringRef& r_string);
31623162
void MCFilesEvalDiskSpace(MCExecContext& ctxt, real64_t& r_result);
31633163
void MCFilesEvalDriverNames(MCExecContext& ctxt, MCStringRef& r_string);
31643164
void MCFilesEvalDrives(MCExecContext& ctxt, MCStringRef& r_string);

‎engine/src/funcs.cpp‎

Lines changed: 13 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -546,7 +546,8 @@ MCFileItems::eval_ctxt(MCExecContext & ctxt, MCExecValue & r_value)
546546
{
547547
MCAutoStringRef t_folder;
548548
bool t_is_detailed = false;
549-
if (m_folder)
549+
bool t_is_utf8 = false;
550+
if (m_folder)
550551
{
551552
if (!ctxt.EvalExprAsStringRef(m_folder, m_files ? EE_FILES_BADFOLDER : EE_FOLDERS_BADFOLDER, &t_folder))
552553
return;
@@ -560,18 +561,26 @@ MCFileItems::eval_ctxt(MCExecContext & ctxt, MCExecValue & r_value)
560561
}
561562
if (!MCStringIsEmpty(*t_kind))
562563
{
563-
if (!MCStringIsEqualToCString(*t_kind, "detailed", kMCStringOptionCompareCaseless))
564+
if (MCStringIsEqualToCString(*t_kind, "detailed", kMCStringOptionCompareCaseless))
565+
{
566+
t_is_detailed = true;
567+
}
568+
else if (MCStringIsEqualToCString(*t_kind, "detailed-utf8", kMCStringOptionCompareCaseless))
569+
{
570+
t_is_detailed = true;
571+
t_is_utf8 = true;
572+
}
573+
else
564574
{
565575
ctxt.LegacyThrow(m_files ? EE_FILES_BADKIND : EE_FOLDERS_BADKIND);
566576
return;
567577
}
568-
t_is_detailed = true;
569578
}
570579
}
571580
}
572581

573582
r_value.type = kMCExecValueTypeStringRef;
574-
MCFilesEvalFileItemsOfDirectory(ctxt, *t_folder, m_files, t_is_detailed, r_value.stringref_value);
583+
MCFilesEvalFileItemsOfDirectory(ctxt, *t_folder, m_files, t_is_detailed, t_is_utf8, r_value.stringref_value);
575584
}
576585

577586

‎engine/src/osspec.h‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -81,7 +81,7 @@ extern bool MCS_resolvepath(MCStringRef p_path, MCStringRef& r_resolved_path);
8181
extern void MCS_getcurdir(MCStringRef& r_path);
8282
/* LEGACY */ extern char *MCS_getcurdir();
8383
extern Boolean MCS_setcurdir(MCStringRef p_path);
84-
extern bool MCS_getentries(MCStringRef p_folder, bool p_files, bool p_detailed, MCListRef& r_list);
84+
extern bool MCS_getentries(MCStringRef p_folder, bool p_files, bool p_detailed, bool p_utf8, MCListRef& r_list);
8585

8686
extern bool MCS_getDNSservers(MCListRef& r_list);
8787
extern Boolean MCS_getdevices(MCStringRef& r_list);

‎engine/src/sysspec.cpp‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -904,6 +904,7 @@ struct MCS_getentries_state
904904
{
905905
bool files;
906906
bool details;
907+
bool utf8;
907908
MCListRef list;
908909
};
909910

@@ -939,7 +940,7 @@ static bool MCS_getentries_callback(void *p_context, const MCSystemFolderEntry *
939940
// SN-2015-01-22: [[ Bug 14412 ]] the detailed files should return
940941
// URL-encoded filenames
941942
MCAutoStringRef t_url_encoded;
942-
if (!MCU_urlencode(*t_normalized, false, &t_url_encoded))
943+
if (!MCU_urlencode(*t_normalized, t_state -> utf8, &t_url_encoded))
943944
return false;
944945

945946
#ifdef _WIN32
@@ -996,6 +997,7 @@ static bool MCS_getentries_callback(void *p_context, const MCSystemFolderEntry *
996997
bool MCS_getentries(MCStringRef p_folder,
997998
bool p_files,
998999
bool p_detailed,
1000+
bool p_utf8,
9991001
MCListRef& r_list)
10001002
{
10011003
MCAutoStringRef t_resolved_folder;
@@ -1014,6 +1016,7 @@ bool MCS_getentries(MCStringRef p_folder,
10141016
MCS_getentries_state t_state;
10151017
t_state.files = p_files;
10161018
t_state.details = p_detailed;
1019+
t_state.utf8 = p_utf8;
10171020
t_state.list = *t_list;
10181021

10191022
// SN-2015-11-09: [[ Bug 16223 ]] Make sure that the list starts with ..

‎tests/lcs/core/files/folders.livecodescript‎

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -46,6 +46,8 @@ on TestFilesOfFolder
4646

4747
TestAssert "list non-current directory (detailed)", the number of items in line 1 of files(tFolder, "detailed") > 1
4848

49+
TestAssert "list non-current directory (detailed-utf8)", the number of items in line 1 of files(tFolder, "detailed-utf8") > 1
50+
4951
local tOldFolder
5052
put the defaultFolder into tOldFolder
5153
set the defaultFolder to tFolder
@@ -68,6 +70,8 @@ on TestFoldersOfFolder
6870

6971
TestAssert "list non-current directory (detailed)", the number of items in line 2 of folders(tFolder, "detailed") > 1
7072

73+
TestAssert "list non-current directory (detailed-utf8)", the number of items in line 2 of folders(tFolder, "detailed-utf8") > 1
74+
7175
local tOldFolder
7276
put the defaultFolder into tOldFolder
7377
set the defaultFolder to tFolder
@@ -126,6 +130,7 @@ on TestEmptyFolder
126130
TestAssert "An empty non-current folder returns '..'", folders(tNewFolder) is ".."
127131
TestAssert "An empty non-current folder returns empty files", files(tNewFolder) is empty
128132
TestAssert "An empty non-current folder returns empty detailed files", files(tNewFolder, "detailed") is empty
133+
TestAssert "An empty non-current folder returns empty detailed-utf8 files", files(tNewFolder, "detailed-utf8") is empty
129134

130135
set the defaultFolder to tNewFolder
131136

@@ -177,4 +182,42 @@ on TestNonExistentFolder
177182
TestAssert "The folders of non-existent folder", folders(tUUID) is empty
178183
TestAssert "The files of non-existent folder", files(tUUID) is empty
179184
TestAssert "The detailed files of non-existent folder", files(tUUID, "detailed") is empty
185+
TestAssert "The detailed-utf8 files of non-existent folder", files(tUUID, "detailed-utf8") is empty
180186
end TestNonExistentFolder
187+
188+
on TestBugfix22213
189+
local tTestFolder
190+
// Create a new, empty folder
191+
put specialFolderPath("temporary") & "/TestUTF8" into tTestFolder
192+
create folder tTestFolder
193+
194+
put empty into url ("file:" & tTestFolder & "/😊.txt")
195+
local tFiles, tFile
196+
put files(tTestFolder, "detailed") into tFiles
197+
put item 1 of line 1 of tFiles into tFile
198+
TestAssertBroken "The detailed files are native encoded before urlEncoding", \
199+
textDecode(urlDecode(tFile), "native") is "😊.txt", "Bug 22213"
200+
201+
put files(tTestFolder, "detailed-utf8") into tFiles
202+
put item 1 of line 1 of tFiles into tFile
203+
TestAssert "The detailed-utf8 files are utf8 encoded before urlEncoding", \
204+
textDecode(urlDecode(tFile), "utf8") is "😊.txt"
205+
206+
delete file (tTestFolder & "/😊.txt")
207+
208+
local tFolders, tFolder
209+
create folder (tTestFolder & "/😊")
210+
put folders(tTestFolder, "detailed") into tFolders
211+
put item 1 of line 2 of tFolders into tFolder
212+
TestAssertBroken "The detailed folders are native encoded before urlEncoding", \
213+
textDecode(urlDecode(tFolder), "native") is "😊", "Bug 22213"
214+
215+
put folders(tTestFolder, "detailed-utf8") into tFolders
216+
TestDiagnostic tFolders
217+
put item 1 of line 2 of tFolders into tFolder
218+
TestAssert "The detailed-utf8 folders are utf8 encoded before urlEncoding", \
219+
textDecode(urlDecode(tFolder), "utf8") is "😊"
220+
221+
delete folder (tTestFolder & "/😊")
222+
delete folder tTestFolder
223+
end TestBugfix22213

0 commit comments

Comments
 (0)